Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-2121

Quay upgrade pods running all workers instead of just database upgrade

    XMLWordPrintable

Details

    Description

      We saw that Quay upgrade pods start all workers and not just database migrations. In a client's case, this had an unfortunate sideeffect where all images were sent to be scanned by Clair v4 instead of just exiting once alembic operations are done. They were testing the upgrade so they used their production environment but without hooking up storage so the security worker ended in a loop where it repeatedly enqueued same layers over and over for scanning and the pod never progressed. This was resolved by deleting the upgrade pods and letting them spin again at which point a correct db version was detected and the instalaltion progressed normally.

      Expected outcome: we only run alembic operations in the upgrade pod without any other workers and then progress and run the main Quay app. The main app should handle all other layer manipulation.

       

      Acceptance criteria:

      • Upgrade stops being a Deployment and becomes a Job instead
      • Upgrade Job use the command line arguments "migrate" and "head"
        • The command ran by the upgrade job should be  'quay-entrypoint.sh migrate head' 
        • This will avoid the dispatch of Gunicorn workers.

      Attachments

        Issue Links

          Activity

            People

              rmarasch@redhat.com Ricardo Maraschini
              rhn-support-ibazulic Ivan Bazulic
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: