Uploaded image for project: 'Red Hat Developer Hub Bugs'
  1. Red Hat Developer Hub Bugs
  2. RHDHBUGS-1142

SPIKE: Investigate why cannot upgrade from RHDH 1.2 to 1.3 (or downgrade from 1.3 to 1.2) fails

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • Core platform
    • None
    • 5
    • False
    • Hide

      None

      Show
      None
    • False

      Description of problem:

      When installing RHDH 1.2 and then upgrading to 1.3  on an EMPTY RHDH instance (no additional plugins or entries in catalog), the backend fails to start with 

      /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1859
          throw new errors.ForwardedError(
                ^

ForwardedError: Plugin 'permission' startup failed; caused by Error: The migration directory is corrupt, the following files are missing: 20240611092136_migrations.js
    at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1859:17
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1858:9
    ... 3 lines matching cause stack trace ...
    at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1949:5) {
  cause: Error: The migration directory is corrupt, the following files are missing: 20240611092136_migrations.js
      at validateMigrationList (/opt/app-root/src/node_modules/knex/lib/migrations/migrate/Migrator.js:567:11)
      at Migrator.latest (/opt/app-root/src/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async migrate (/opt/app-root/src/node_modules/@janus-idp/backstage-plugin-rbac-backend/dist/index.cjs.js:288:5)
      at async PolicyBuilder.build (/opt/app-root/src/node_modules/@janus-idp/backstage-plugin-rbac-backend/dist/index.cjs.js:3713:5)
      at async Object.init [as func] (/opt/app-root/src/node_modules/@janus-idp/backstage-plugin-rbac-backend/dist/index.cjs.js:3818:11)
      at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1858:9
      at async Promise.all (index 6)
      at async BackendInitializer.doStart_fn (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1819:3)
      at async BackendInitializer.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1675:5)
      at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1949:5)
}

Node.js v20.14.0

      When rolling back from 1.3 (1.3-50 image using v1.27.7 of backstage) back to 1.2.2 (1.2-132 image) on an EMPTY RHDH instance (no additional plugins or entries in catalog), you get the following error:

      /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1806
          throw new errors.ForwardedError(
                ^

ForwardedError: Plugin 'scaffolder' startup failed; caused by Error: The migration directory is corrupt, the following files are missing: 20240401213200_workspace.js
    at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1806:17
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1805:9
    ... 3 lines matching cause stack trace ...
    at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1894:5) {
  cause: Error: The migration directory is corrupt, the following files are missing: 20240401213200_workspace.js
      at validateMigrationList (/opt/app-root/src/node_modules/knex/lib/migrations/migrate/Migrator.js:567:11)
      at Migrator.latest (/opt/app-root/src/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async DatabaseTaskStore.runMigrations (/opt/app-root/src/node_modules/@backstage/plugin-scaffolder-backend/dist/cjs/router-CDFi_apW.cjs.js:1598:7)
      at async DatabaseTaskStore.create (/opt/app-root/src/node_modules/@backstage/plugin-scaffolder-backend/dist/cjs/router-CDFi_apW.cjs.js:1558:5)
      at async Object.createRouter (/opt/app-root/src/node_modules/@backstage/plugin-scaffolder-backend/dist/cjs/router-CDFi_apW.cjs.js:3502:31)
      at async Object.init [as func] (/opt/app-root/src/node_modules/@backstage/plugin-scaffolder-backend/dist/alpha.cjs.js:155:26)
      at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1805:9
      at async Promise.all (index 3)
      at async BackendInitializer.doStart_fn (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1766:3)
      at async BackendInitializer.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1626:5)
      at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1894:5)
}

Node.js v20.14.0

      Only way to get backend to start for me was to delete the postgresql PVC and regenerate a new one

      Prerequisites (if any, like setup, operators/versions):

      Steps to Reproduce

      1. Install RHDH 1.2.2 (registry.redhat.io/rhdh/rhdh-hub-rhel9@sha256:9709e056f194a24c809dd82b4ab10aa05a9c31574e68ae8d1a5e91de19237045)
      2. Upgrade to RHDH 1.3 (change the image to 1.3-50 https://quay.io/repository/rhdh/rhdh-hub-rhel9/manifest/sha256:59269c27c8f14918e267ef91f59091bf1375432c3c85ea0a18d216ca561717d9)
      3. Observer error

      Alternatively, for the downgrade

      1. Install RHDH 1.3 (use 1.3-50 https://quay.io/repository/rhdh/rhdh-hub-rhel9/manifest/sha256:59269c27c8f14918e267ef91f59091bf1375432c3c85ea0a18d216ca561717d9)
      2. Downgrade to RHDH 1.3 (use 1.2.2 registry.redhat.io/rhdh/rhdh-hub-rhel9@sha256:9709e056f194a24c809dd82b4ab10aa05a9c31574e68ae8d1a5e91de19237045)
      3. Observe error

      Actual results:

      RHDH Pod fails to start after upgrade/downgrade

      Expected results:

      RHDH Pod starts up properly after upgrade/downgrade

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Build Details:

      1.3-50 build:

      RHDH Version: 1.3.0
      Backstage Version: 1.27.7
      Upstream: https://github.com/janus-idp/backstage-showcase/tree/main @ 1e002edd
      Midstream: https://gitlab.cee.redhat.com/rhidp/rhdh/-/commits/rhdh-1-rhel-9 @ c139fd50
      Build Time: 2024-08-01T02:36:08Z

      RHDH 1.22:

      RHDH Version: 1.2.2
      Backstage Version: 1.26.5
      Upstream: janus-idp/backstage-showcase 1.2.x @ c14d6dac
      Midstream: gitlab.cee.redhat.com/rhidp/rhdh rhdh-1.2-rhel-9 @ 01df3afc
      Build Time: 2024-07-26T10:06:37Z

      Additional info (Such as Logs, Screenshots, etc):

              Unassigned Unassigned
              rh-ee-frkong Frank Kong
              RHIDP - Cope
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: