Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-18274

6.13 upgrade fails with deadlock detected when migrating pulpcore database

XMLWordPrintable

    • Important
    • None

      Description of problem:

      Some customers are hitting below error when upgrading 6.13:

      ~~~
      2023-05-11 18:05:53 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: raise dj_exc_value.with_traceback(traceback) from exc_value
      2023-05-11 18:05:53 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
      2023-05-11 18:05:53 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: return self.cursor.execute(sql, params)
      2023-05-11 18:05:53 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: django.db.utils.OperationalError: deadlock detected
      2023-05-11 18:05:53 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: DETAIL: Process 107920 waits for AccessExclusiveLock on relation 3049568 of database 2369498; blocked by process 102571.
      2023-05-11 18:05:53 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Process 102571 waits for AccessShareLock on relation 2369756 of database 2369498; blocked by process 107920.
      2023-05-11 18:05:53 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: HINT: See server log for query details.
      2023-05-11 18:05:53 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
      2023-05-11 18:05:53 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
      2023-05-11 18:05:53 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Evaluated in 18.01 seconds
      ~~~

      Similar to this upstream issue:

      https://community.theforeman.org/t/katello-4-7-0-rc2-upgrade/31451

      Version-Release number of selected component (if applicable):

      How reproducible:

      Hard

      Steps to Reproduce:

      It only happens on the customer side.

      Actual results:

      Upgrade fails with the pulp migration error.

      Expected results:

      Upgrade should have no error.

      Additional info:

      The workaround from the upstream is valid:

      ~~~
      foreman-maintain service stop
      systemctl start postgresql
      systemctl start redis
      PULP_SETTINGS=/etc/pulp/settings.py DJANGO_SETTINGS_MODULE=pulpcore.app.settings pulpcore-manager migrate --noinput
      satellite-installer
      ~~~

            egolov@redhat.com Evgeni Golov
            satellite-jira-automation@redhat.com Satellite Jira-Automation
            RH Bugzilla Integration RH Bugzilla Integration
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: