Uploaded image for project: 'Red Hat Developer Hub Bugs'
  1. Red Hat Developer Hub Bugs
  2. RHDHBUGS-1582

RHDH fails to start with "MigrationLocked: Plugin 'auth' startup failed; caused by MigrationLocked: Migration table is already locked" error.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Normal Normal
    • None
    • None
    • Helm Chart
    • None
    • 1
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide
      = Deployment fails with error: Migration table is already locked

      If you have 1 or more pre-existing Backstage instances on your cluster, you may encounter an error when deploying a new instance, even if in another namespace, or when or updating to the latest 1.1.z chart.

      This may be caused if existing charts or database pods are incompatible with the new deployment.

      Workaround: save any existing configurations (configmaps, chart values), then delete existing deployments. Deploy the latest 1.1.z chart.
      Show
      = Deployment fails with error: Migration table is already locked If you have 1 or more pre-existing Backstage instances on your cluster, you may encounter an error when deploying a new instance, even if in another namespace, or when or updating to the latest 1.1.z chart. This may be caused if existing charts or database pods are incompatible with the new deployment. Workaround: save any existing configurations (configmaps, chart values), then delete existing deployments. Deploy the latest 1.1.z chart.
    • Known Issue
    • Proposed
    • 1
    • RHDH Core Team 3256

      Description of problem:

      When installing Developer Hub (v.1.1.2) with helm charts, the out-of-the-box setup fails to start with the following error:

      Loading config from MergedConfigSource{FileConfigSource{path="/opt/app-root/src/app-config.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.production.yaml"}, FileConfigSource{path="/opt/app-root/src/dynamic-plugins-root/app-config.dynamic-plugins.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config-from-configmap.yaml"}, EnvConfigSource{count=1}} skipping '/opt/app-root/src/dynamic-plugins-root/app-config.dynamic-plugins.yaml' since it is not a directory failed to load dynamic plugin manifest from '/opt/app-root/src/dynamic-plugins-root/lost+found' [Error: ENOENT: no such file or directory, open '/opt/app-root/src/dynamic-plugins-root/lost+found/package.json'] { errno: -2, code: 'ENOENT', syscall: 'open', path: '/opt/app-root/src/dynamic-plugins-root/lost+found/package.json' } 

[2m2024-04-19T11:45:11.356Z[22m [34mbackstage

 [32minfo

 Found 2 new secrets in config that will be redacted 

[2m2024-04-19T11:45:11.358Z[22m [34mbackstage

 [32minfo

 skipping '/opt/app-root/src/dynamic-plugins-root/app-config.dynamic-plugins.yaml' since it is not a directory 

[2m2024-04-19T11:45:11.365Z[22m [34mbackstage

 [31merror

 failed to load dynamic plugin manifest from '/opt/app-root/src/dynamic-plugins-root/lost+found' ENOENT: no such file or directory, open '/opt/app-root/src/dynamic-plugins-root/lost+found/package.json' [36merrno

=-2 [36mcode

=ENOENT [36msyscall

=open [36mpath

=/opt/app-root/src/dynamic-plugins-root/lost+found/package.json [36mstack

=Error: ENOENT: no such file or directory, open '/opt/app-root/src/dynamic-plugins-root/lost+found/package.json' 

[2m2024-04-19T11:45:11.507Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-azure-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-azure-dynamic-0.1.5' 

[2m2024-04-19T11:45:11.558Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-bitbucket-cloud-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-bitbucket-cloud-dynamic-0.1.3' 

[2m2024-04-19T11:45:11.587Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-bitbucket-server-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-bitbucket-server-dynamic-0.1.3' 

[2m2024-04-19T11:45:11.643Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-gerrit-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-gerrit-dynamic-0.1.5' 

[2m2024-04-19T11:45:11.790Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-github-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-github-dynamic-0.2.3' 

[2m2024-04-19T11:45:11.952Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-gitlab-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-gitlab-dynamic-0.2.16' 

[2m2024-04-19T11:45:12.656Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'backstage-plugin-techdocs-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-techdocs-backend-dynamic-1.9.6' 

[2m2024-04-19T11:45:12.657Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-quay-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-quay-dynamic-1.3.5/alpha' 

[2m2024-04-19T11:45:12.718Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-regex-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-regex-dynamic-1.3.5/alpha' 

[2m2024-04-19T11:45:12.806Z[22m [34mbackstage

 [32minfo

 loaded dynamic backend plugin 'roadiehq-scaffolder-backend-module-utils-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-scaffolder-backend-module-utils-dynamic-1.13.6' 

[2m2024-04-19T11:45:12.822Z[22m [34mrootHttpRouter

 [32minfo

 Listening on :7007 

[2m2024-04-19T11:45:12.846Z[22m [34mauth

 [32minfo

 Enabled Provider Factories : {} 

[2m2024-04-19T11:45:12.867Z[22m [34mscalprum

 [32minfo

 Loaded dynamic frontend plugin 'backstage-plugin-techdocs' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-techdocs-1.10.0' 

[2m2024-04-19T11:45:12.868Z[22m [34mscalprum

 [32minfo

 Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-dynamic-plugins-info' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-dynamic-plugins-info-1.0.2' 

[2m2024-04-19T11:45:12.871Z[22m [34mproxy

 [32minfo

 [HPM] Proxy created: /quay/api -> https://quay.io/ 

[2m2024-04-19T11:45:12.872Z[22m [34mproxy

 [32minfo

 [HPM] Proxy rewrite rule created: "^/api/proxy/quay/api/?" ~> "/" 

[2m2024-04-19T11:45:12.872Z[22m [34mproxy

 [33mwarn

 skipped configuring /sonarqube due to Proxy target for route "/sonarqube" must be a string, but is of type undefined 

[2m2024-04-19T11:45:12.873Z[22m [34mproxy

 [33mwarn

 skipped configuring /jenkins/api due to Proxy target for route "/jenkins/api" must be a string, but is of type undefined 

[2m2024-04-19T11:45:12.873Z[22m [34mproxy

 [33mwarn

 skipped configuring /jira/api due to Proxy target for route "/jira/api" must be a string, but is of type undefined 

[2m2024-04-19T11:45:12.873Z[22m [34mproxy

 [33mwarn

 skipped configuring /acr/api due to Proxy target for route "/acr/api" must be a string, but is of type undefined 

[2m2024-04-19T11:45:12.873Z[22m [34mproxy

 [33mwarn

 skipped configuring /jfrog-artifactory/api due to Proxy target for route "/jfrog-artifactory/api" must be a string, but is of type undefined 

[2m2024-04-19T11:45:12.873Z[22m [34mproxy

 [32minfo

 [HPM] Proxy created: /pagerduty -> https://api.pagerduty.com 

[2m2024-04-19T11:45:12.873Z[22m [34mproxy

 [32minfo

 [HPM] Proxy rewrite rule created: "^/api/proxy/pagerduty/?" ~> "/" 

[2m2024-04-19T11:45:12.874Z[22m [34mproxy

 [33mwarn

 skipped configuring /dynatrace due to Proxy target for route "/dynatrace" must be a string, but is of type undefined 

[2m2024-04-19T11:45:12.874Z[22m [34mproxy

 [33mwarn

 skipped configuring /nexus-repository-manager due to Proxy target for route "/nexus-repository-manager" must be a string, but is of type undefined 

[2m2024-04-19T11:45:12.880Z[22m [34mtechdocs

 [32minfo

 Creating Local publisher for TechDocs 

[2m2024-04-19T11:45:12.881Z[22m [34mbackstage

 [32minfo

 Resolving '@backstage/plugin-techdocs-backend/package.json' in the dynamic backend plugins 

[2m2024-04-19T11:45:12.919Z[22m [34mauth

 [32minfo

 Configuring "database" as KeyStore provider 

[2m2024-04-19T11:45:12.927Z[22m [34mscaffolder

 [32minfo

 Starting scaffolder with the following actions enabled quay:create-repository, regex:replace, publish:azure, publish:bitbucketCloud, bitbucket:pipelines:run, publish:bitbucketServer, publish:bitbucketServer:pull-request, publish:gerrit, publish:gerrit:review, github:actions:dispatch, github:autolinks:create, github:deployKey:create, github:environment:create, github:issues:label, github:repo:create, github:repo:push, github:webhook, publish:github, publish:github:pull-request, gitlab:group:ensureExists, gitlab:issues:create, gitlab:projectAccessToken:create, gitlab:projectDeployToken:create, gitlab:projectVariable:create, gitlab:repo:push, publish:gitlab, publish:gitlab:merge-request, roadiehq:utils:zip, roadiehq:utils:sleep, roadiehq:utils:fs:write, roadiehq:utils:fs:append, roadiehq:utils:json:merge, roadiehq:utils:merge, roadiehq:utils:fs:parse, roadiehq:utils:fs:replace, roadiehq:utils:serialize:yaml, roadiehq:utils:serialize:json, roadiehq:utils:jsonata, roadiehq:utils:jsonata:yaml:transform, roadiehq:utils:jsonata:json:transform, fetch:plain, fetch:plain:file, fetch:template, debug:log, debug:wait, catalog:register, catalog:fetch, catalog:write, fs:delete, fs:rename 

[2m2024-04-19T11:45:12.930Z[22m [34msearch

 [32minfo

 Added DefaultCatalogCollatorFactory collator factory for type software-catalog 

[2m2024-04-19T11:45:12.931Z[22m [34msearch

 [32minfo

 Added DefaultTechDocsCollatorFactory collator factory for type techdocs 

[2m2024-04-19T11:45:12.931Z[22m [34msearch

 [32minfo

 Starting all scheduled search tasks. 

[2m2024-04-19T11:45:13.748Z[22m [34mapp

 [32minfo

 Serving static app content from /opt/app-root/src/packages/app/dist 

[2m2024-04-19T11:45:13.808Z[22m [34mapp

 [32minfo

 Injecting env config into 1378.bd850c85.chunk.js 

[2m2024-04-19T11:45:13.860Z[22m [34mcatalog

 [32minfo

 Performing database migration Can't take lock to run migrations: Migration table is already locked If you are sure migrations are not running you can release the lock manually by running 'knex migrate:unlock' /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1793 throw new errors.ForwardedError( ^ MigrationLocked: Plugin 'auth' startup failed; caused by MigrationLocked: Migration table is already locked at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1793:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1792:9 at async Promise.all (index 7) at async BackendInitializer.doStart_fn (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1757:3) at async BackendInitializer.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1611:5) at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1881:5) { cause: MigrationLocked: Migration table is already locked at /opt/app-root/src/node_modules/knex/lib/migrations/migrate/Migrator.js:343:13 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Migrator._runBatch (/opt/app-root/src/node_modules/knex/lib/migrations/migrate/Migrator.js:359:7) } Node.js v18.19.1

      The same cluster was used to create other RHDH instances with the Operator but in different namespaces. 
      Slack thread: https://redhat-internal.slack.com/archives/C04CUSD4JSG/p1713527319304689

       

      Prerequisites (if any, like setup, operators/versions):

      Steps to Reproduce

      1. get an OpenShift cluster (used v4.15)
      2. setup rhdh with helm charts
      curl -sSLO https://raw.githubusercontent.com/rhdh-bot/openshift-helm-charts/redhat-developer-hub-1.1-104-CI/installation/install.sh && chmod +x install.sh && ./install.sh 1.1-104-CI --namespace rhdh-1-1-104-ci --chartrepo
      1. pod fails to start and logs show the error above

      Actual results:

      pod fails to start and logs show the error above

      Expected results:

      the starts runs without errors

      Reproducibility (Always/Intermittent/Only Once):

      Build Details:

      version 1.1.2

      Additional info (Such as Logs, Screenshots, etc):

              rhn-support-cdaley Corey Daley (Inactive)
              rh-ee-abarbaro Alessandro Barbarossa
              RHDH Install
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: