-
Bug
-
Resolution: Done
-
Major
-
1.7.0, 1.7.1
-
None
-
2
-
False
-
-
False
-
-
Bug Fix
-
Proposed
-
-
-
RHDH Install 3279, RHDH Install 3280, RHDH Install 3281
-
Important
Bug Summary
After upgrading from Orchestrator Operator + RHDH 1.6 to reinstalling RHDH via Helm with Orchestrator enabled, the rhdh-developer-hub pod fails its readiness probe.
Symptoms:
Pod status: Running, but not Ready.
backstage-backend container shows repeated log messages:
Plugin initialization in progress, still initializing: 'healthcheck', 'app', 'catalog', 'proxy', ...
Kubernetes reports:
Readiness probe failed: HTTP probe failed with statuscode: 503
Environment
Namespace: rhdh-chart Image: quay.io/rhdh/rhdh-hub-rhel9:latest Postgres host: rhdh-postgresql (auto-provisioned by chart)
Cause:
Missing Network policies – orchestrator plugin requires some network policies to allow communication between the sonataflow components and rhdh.
In an upgrade scenario, the user will be reusing the Openshift Serverless Logic and Openshift Serverless Installations they have from the previous set up. Because they already have it, there is no need for the Backstage chart to install the sonataflow resources again. Therfore, the current logic is flawed here.
When installing the Helm Chart with these values:
orchestrator: enabled: true serverlessLogicOperator: enabled: false serverlessOperator: enabled: false # optional
The Helm Chart has a flow control to not install the network policies. These are the fault and should be fixed to allow the installation of the network policies, even if the serverlessLogicOperator and serverlessOperator values are false.
Workaround:
manually add the network policies to the rhdh namespace.
Prerequisites (if any, like setup, operators/versions):
Steps to Reproduce:
1. Install Orchestrator Operator v1.6
2. Follow the upgrade guide steps to upgrade to RHDH v1.7 via the helm chart.
3. Make sure to perserve te old sonataflow resources and follow the steps in the upgrade guide.
3. Delete rhdh v1.6
4. Install rhdh v1.7 per the upgrade guide via the helm chart
Actual results:
Backstage pod fails to finish, getting a 503 error
Expected results:
Backstage pod is OK
Reproducibility (Always/Intermittent/Only Once):
100%
Build Details:
Additional info (Such as Logs, Screenshots, etc):
The place that needs a code fix:
https://github.com/redhat-developer/rhdh-chart/blob/main/charts/backstage/templates/network-policies.yaml
more logs:
eshalev orchestrator-demo (main) $ oc describe pod rhdh-developer-hub-57465c987-tfn72 Name: rhdh-developer-hub-57465c987-tfn72 Namespace: rhdh Priority: 0 Service Account: default Node: ip-10-0-51-212.eu-west-1.compute.internal/10.0.51.212 Start Time: Mon, 08 Sep 2025 15:01:45 +0300 Labels: app.kubernetes.io/component=backstage app.kubernetes.io/instance=rhdh app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=developer-hub helm.sh/chart=upstream-2.6.1 pod-template-hash=57465c987 Annotations: checksum/app-config: 035e7c1d53b826d4189c0dbd9e1867575e23b07801f628c51e8de563c296fa28 checksum/dynamic-plugins: 1d804977083deba2f7b9eabb6aa0be3ae29ce14612d6a11250403b436cfe08c2 k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.130.0.44/23"],"mac_address":"0a:58:0a:82:00:2c","gateway_ips":["10.130.0.1"],"routes":[{"dest":"10.128.0.0... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.130.0.44" ], "mac": "0a:58:0a:82:00:2c", "default": true, "dns": {} }] openshift.io/scc: restricted-v2 seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Running SeccompProfile: RuntimeDefault IP: 10.130.0.44 IPs: IP: 10.130.0.44 Controlled By: ReplicaSet/rhdh-developer-hub-57465c987 Init Containers: install-dynamic-plugins: Container ID: cri-o://01f75632e27ef4ed9addadc7c38436e31d9e3171ff4c3962273b7fbe02148caf Image: quay.io/rhdh/rhdh-hub-rhel9:latest Image ID: quay.io/rhdh/rhdh-hub-rhel9@sha256:1833cd22478ed53a99e08596b97c892559120b602a7cadcfe4be2a3ecf74a91a Port: <none> Host Port: <none> SeccompProfile: RuntimeDefault Command: ./install-dynamic-plugins.sh /dynamic-plugins-root State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 08 Sep 2025 15:01:51 +0300 Finished: Mon, 08 Sep 2025 15:04:44 +0300 Ready: True Restart Count: 0 Limits: cpu: 1 ephemeral-storage: 5Gi memory: 2560Mi Requests: cpu: 250m ephemeral-storage: 5Gi memory: 256Mi Environment: NPM_CONFIG_USERCONFIG: /opt/app-root/src/.npmrc.dynamic-plugins MAX_ENTRY_SIZE: 30000000 Mounts: /dynamic-plugins-root from dynamic-plugins-root (rw) /opt/app-root/src/.config/containers from dynamic-plugins-registry-auth (ro) /opt/app-root/src/.npm/_cacache from npmcacache (rw) /opt/app-root/src/.npmrc.dynamic-plugins from dynamic-plugins-npmrc (ro,path=".npmrc") /opt/app-root/src/dynamic-plugins.yaml from dynamic-plugins (ro,path="dynamic-plugins.yaml") /tmp from temp (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58srb (ro) Containers: backstage-backend: Container ID: cri-o://394d8f1732855fb641f982019033e8c3d5f6fdc4885673a6f3ca261a2c771c7a Image: quay.io/rhdh/rhdh-hub-rhel9:latest Image ID: quay.io/rhdh/rhdh-hub-rhel9@sha256:1833cd22478ed53a99e08596b97c892559120b602a7cadcfe4be2a3ecf74a91a Port: 7007/TCP Host Port: 0/TCP SeccompProfile: RuntimeDefault Args: --config dynamic-plugins-root/app-config.dynamic-plugins.yaml --config /opt/app-root/src/app-config-from-configmap.yaml State: Running Started: Mon, 08 Sep 2025 15:04:47 +0300 Ready: False Restart Count: 0 Limits: cpu: 1 ephemeral-storage: 5Gi memory: 2560Mi Requests: cpu: 250m ephemeral-storage: 5Gi memory: 1Gi Liveness: http-get http://:backend/.backstage/health/v1/liveness delay=0s timeout=4s period=10s #success=1 #failure=3 Readiness: http-get http://:backend/.backstage/health/v1/readiness delay=0s timeout=4s period=10s #success=2 #failure=3 Startup: http-get http://:backend/.backstage/health/v1/liveness delay=30s timeout=4s period=20s #success=1 #failure=3 Environment: APP_CONFIG_backend_listen_port: 7007 POSTGRES_HOST: rhdh-postgresql POSTGRES_PORT: 5432 POSTGRES_USER: bn_backstage POSTGRES_PASSWORD: <set to the key 'password' in secret 'rhdh-postgresql'> Optional: false BACKEND_SECRET: <set to the key 'backend-secret' in secret 'rhdh-auth'> Optional: false POSTGRESQL_ADMIN_PASSWORD: <set to the key 'postgres-password' in secret 'rhdh-postgresql'> Optional: false Mounts: /opt/app-root/src/app-config-from-configmap.yaml from backstage-app-config (rw,path="app-config.yaml") /opt/app-root/src/dynamic-plugins-root from dynamic-plugins-root (rw) /tmp from temp (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58srb (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: dynamic-plugins-root: Type: EphemeralVolume (an inline specification for a volume that gets created and deleted with the pod) StorageClass: Volume: Labels: <none> Annotations: <none> Capacity: Access Modes: VolumeMode: Filesystem dynamic-plugins: Type: ConfigMap (a volume populated by a ConfigMap) Name: rhdh-dynamic-plugins Optional: true dynamic-plugins-npmrc: Type: Secret (a volume populated by a Secret) SecretName: rhdh-dynamic-plugins-npmrc Optional: true dynamic-plugins-registry-auth: Type: Secret (a volume populated by a Secret) SecretName: rhdh-dynamic-plugins-registry-auth Optional: true npmcacache: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> temp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> backstage-app-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: rhdh-developer-hub-app-config Optional: false kube-api-access-58srb: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 3m50s default-scheduler 0/4 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "rhdh-developer-hub-57465c987-tfn72-dynamic-plugins-root". preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling. Normal Scheduled 3m45s default-scheduler Successfully assigned rhdh/rhdh-developer-hub-57465c987-tfn72 to ip-10-0-51-212.eu-west-1.compute.internal Normal SuccessfulAttachVolume 3m42s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c2582d69-6c6f-42fb-8ca9-7b8cde75c368" Normal AddedInterface 3m40s multus Add eth0 [10.130.0.44/23] from ovn-kubernetes Normal Pulling 3m40s kubelet Pulling image "quay.io/rhdh/rhdh-hub-rhel9:latest" Normal Pulled 3m39s kubelet Successfully pulled image "quay.io/rhdh/rhdh-hub-rhel9:latest" in 799ms (799ms including waiting). Image size: 4179534415 bytes. Normal Created 3m39s kubelet Created container: install-dynamic-plugins Normal Started 3m39s kubelet Started container install-dynamic-plugins Normal Pulling 45s kubelet Pulling image "quay.io/rhdh/rhdh-hub-rhel9:latest" Normal Pulled 44s kubelet Successfully pulled image "quay.io/rhdh/rhdh-hub-rhel9:latest" in 944ms (944ms including waiting). Image size: 4179534415 bytes. Normal Created 43s kubelet Created container: backstage-backend Normal Started 43s kubelet Started container backstage-backend Warning Unhealthy 1s kubelet Readiness probe failed: HTTP probe failed with statuscode: 503 eshalev orchestrator-demo (main) $ oc logs rhdh-developer-hub-57465c987-tfn72 Defaulted container "backstage-backend" out of: backstage-backend, install-dynamic-plugins (init) 2025-09-08T12:04:54.883Z developer-hub-init info Starting Developer Hub backend (node:1) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time. (Use `node --trace-warnings ...` to show where the warning was created) Loading config from MergedConfigSource{FileConfigSource{path="/opt/app-root/src/app-config.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.production.yaml"}, FileConfigSource{path="/opt/app-root/src/dynamic-plugins-root/app-config.dynamic-plugins.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config-from-configmap.yaml"}, EnvConfigSource{count=1}} skipping '/opt/app-root/src/dynamic-plugins-root/app-config.dynamic-plugins.yaml' since it is not a directory skipping 'lost+found' system directory 2025-09-08T12:05:10.094Z backstage info Found 2 new secrets in config that will be redacted 2025-09-08T12:05:10.097Z backstage info skipping '/opt/app-root/src/dynamic-plugins-root/app-config.dynamic-plugins.yaml' since it is not a directory 2025-09-08T12:05:10.302Z backstage info loaded dynamic backend plugin 'backstage-community-plugin-scaffolder-backend-module-quay-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-community-plugin-scaffolder-backend-module-quay-dynamic-2.9.1' 2025-09-08T12:05:10.473Z backstage info loaded dynamic backend plugin 'backstage-community-plugin-scaffolder-backend-module-regex-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-community-plugin-scaffolder-backend-module-regex-dynamic-2.7.0' 2025-09-08T12:05:10.692Z backstage info Resolving '@backstage/plugin-notifications-backend/package.json' in the dynamic backend plugins 2025-09-08T12:05:10.694Z backstage info Resolved '@backstage/plugin-notifications-backend' at /opt/app-root/src/dynamic-plugins-root/backstage-plugin-notifications-backend-dynamic-0.5.6/node_modules/@backstage/plugin-notifications-backend/package.json 2025-09-08T12:05:10.732Z backstage info loaded dynamic backend plugin 'backstage-plugin-notifications-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-notifications-backend-dynamic-0.5.6' 2025-09-08T12:05:10.972Z backstage info loaded dynamic backend plugin 'backstage-plugin-signals-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-signals-backend-dynamic-0.3.4' 2025-09-08T12:05:13.165Z backstage info loaded dynamic backend plugin 'backstage-plugin-techdocs-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-techdocs-backend-dynamic-2.0.2' 2025-09-08T12:05:13.364Z backstage info Resolving '@red-hat-developer-hub/backstage-plugin-adoption-insights-backend/package.json' in the dynamic backend plugins 2025-09-08T12:05:13.365Z backstage info Resolved '@red-hat-developer-hub/backstage-plugin-adoption-insights-backend' at /opt/app-root/src/dynamic-plugins-root/red-hat-developer-hub-backstage-plugin-adoption-insights-backend-dynamic-0.2.2/node_modules/@red-hat-developer-hub/backstage-plugin-adoption-insights-backend/package.json 2025-09-08T12:05:13.453Z backstage info loaded dynamic backend plugin 'red-hat-developer-hub-backstage-plugin-adoption-insights-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/red-hat-developer-hub-backstage-plugin-adoption-insights-backend-dynamic-0.2.2' 2025-09-08T12:05:13.622Z backstage info loaded dynamic backend plugin 'red-hat-developer-hub-backstage-plugin-catalog-backend-module-marketplace-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/red-hat-developer-hub-backstage-plugin-catalog-backend-module-marketplace-dynamic-0.4.4' 2025-09-08T12:05:13.973Z backstage info loaded dynamic backend plugin 'red-hat-developer-hub-backstage-plugin-marketplace-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/red-hat-developer-hub-backstage-plugin-marketplace-backend-dynamic-0.7.3' 2025-09-08T12:05:16.088Z backstage info loaded dynamic backend plugin '@redhat/backstage-plugin-orchestrator-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/redhat-backstage-plugin-orchestrator-backend-dynamic-1.6.0' 2025-09-08T12:05:17.132Z backstage info loaded dynamic backend plugin '@redhat/backstage-plugin-scaffolder-backend-module-orchestrator-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/redhat-backstage-plugin-scaffolder-backend-module-orchestrator-dynamic-1.6.0' 2025-09-08T12:05:17.169Z rootHttpRouter info Listening on :7007 2025-09-08T12:05:17.173Z backstage info Plugin initialization started: 'healthcheck', 'app', 'catalog', 'proxy', 'scaffolder', 'search', 'events', 'permission', 'auth', 'dynamic-plugins-info', 'scalprum', 'licensed-users-info', 'notifications', 'signals', 'adoption-insights', 'extensions', 'orchestrator', 'techdocs' type="initialization" 2025-09-08T12:05:18.174Z backstage info Plugin initialization in progress, still initializing: 'healthcheck', 'app', 'catalog', 'proxy', 'scaffolder', 'search', 'events', 'permission', 'auth', 'dynamic-plugins-info', 'scalprum', 'licensed-users-info', 'notifications', 'signals', 'adoption-insights', 'extensions', 'orchestrator', 'techdocs' type="initialization" 2025-09-08T12:05:19.175Z backstage info Plugin initialization in progress, still initializing: 'healthcheck', 'app', 'catalog', 'proxy', 'scaffolder', 'search', 'events', 'permission', 'auth', 'dynamic-plugins-info', 'scalprum', 'licensed-users-info', 'notifications', 'signals', 'adoption-insights', 'extensions', 'orchestrator', 'techdocs' type="initialization" 2025-09-08T12:05:21.176Z backstage info Plugin initialization in progress, still initializing: 'healthcheck', 'app', 'catalog', 'proxy', 'scaffolder', 'search', 'events', 'permission', 'auth', 'dynamic-plugins-info', 'scalprum', 'licensed-users-info', 'notifications', 'signals', 'adoption-insights', 'extensions', 'orchestrator', 'techdocs' type="initialization" 2025-09-08T12:05:24.177Z backstage info Plugin initialization in progress, still initializing: 'healthcheck', 'app', 'catalog', 'proxy', 'scaffolder', 'search', 'events', 'permission', 'auth', 'dynamic-plugins-info', 'scalprum', 'licensed-users-info', 'notifications', 'signals', 'adoption-insights', 'extensions', 'orchestrator', 'techdocs' type="initialization" 2025-09-08T12:05:29.178Z backstage info Plugin initialization in progress, still initializing: 'healthcheck', 'app', 'catalog', 'proxy', 'scaffolder', 'search', 'events', 'permission', 'auth', 'dynamic-plugins-info', 'scalprum', 'licensed-users-info', 'notifications', 'signals', 'adoption-insights', 'extensions', 'orchestrator', 'techdocs' type="initialization" 2025-09-08T12:05:29.863Z rootHttpRouter info [2025-09-08T12:05:29.863Z] "GET /.backstage/health/v1/liveness HTTP/1.1" 200 15 "-" "kube-probe/1.30" type="incomingRequest" date="2025-09-08T12:05:29.863Z" method="GET" url="/.backstage/health/v1/liveness" status=200 httpVersion="1.1" userAgent="kube-probe/1.30" contentLength=15 trace_id="44e249b598f392b0874d32b6a8326b6f" span_id="1a1482fa8432e03e" trace_flags="01" 2025-09-08T12:05:29.871Z rootHttpRouter info [2025-09-08T12:05:29.871Z] "GET /.backstage/health/v1/readiness HTTP/1.1" 503 58 "-" "kube-probe/1.30" type="incomingRequest" date="2025-09-08T12:05:29.871Z" method="GET" url="/.backstage/health/v1/readiness" status=503 httpVersion="1.1" userAgent="kube-probe/1.30" contentLength=58 trace_id="9cad5e44e77876427e94220a7f07ebcd" span_id="e416c65b5a9b23a5" trace_flags="01" 2025-09-08T12:05:37.178Z backstage info Plugin initialization in progress, still initializing: 'healthcheck', 'app', 'catalog', 'proxy', 'scaffolder', 'search', 'events', 'permission', 'auth', 'dynamic-plugins-info', 'scalprum', 'licensed-users-info', 'notifications', 'signals', 'adoption-insights', 'extensions', 'orchestrator', 'techdocs' type="initialization" 2025-09-08T12:05:39.840Z rootHttpRouter info [2025-09-08T12:05:39.840Z] "GET /.backstage/health/v1/liveness HTTP/1.1" 200 15 "-" "kube-probe/1.30" type="incomingRequest" date="2025-09-08T12:05:39.840Z" method="GET" url="/.backstage/health/v1/liveness" status=200 httpVersion="1.1" userAgent="kube-probe/1.30" contentLength=15 trace_id="4a951563344a4a268ee2bfe053d38fd2" span_id="96d1fb6b6df00fcf" trace_flags="01" 2025-09-08T12:05:39.844Z rootHttpRouter info [2025-09-08T12:05:39.844Z] "GET /.backstage/health/v1/readiness HTTP/1.1" 503 58 "-" "kube-probe/1.30" type="incomingRequest" date="2025-09-08T12:05:39.844Z" method="GET" url="/.backstage/health/v1/readiness" status=503 httpVersion="1.1" userAgent="kube-probe/1.30" contentLength=58 trace_id="cc7ab311bd91a7fcf7fa83b406cb175c" span_id="cb57c2ad8fb8c9ce" trace_flags="01" 2025-09-08T12:05:49.841Z rootHttpRouter info [2025-09-08T12:05:49.841Z] "GET /.backstage/health/v1/liveness HTTP/1.1" 200 15 "-" "kube-probe/1.30" type="incomingRequest" date="2025-09-08T12:05:49.841Z" method="GET" url="/.backstage/health/v1/liveness" status=200 httpVersion="1.1" userAgent="kube-probe/1.30" contentLength=15 trace_id="31edf58174e2c5879d5f5db0085483c3" span_id="7782161d86a193ac" trace_flags="01" 2025-09-08T12:05:49.846Z rootHttpRouter info [2025-09-08T12:05:49.846Z] "GET /.backstage/health/v1/readiness HTTP/1.1" 503 58 "-" "kube-probe/1.30" type="incomingRequest" date="2025-09-08T12:05:49.846Z" method="GET" url="/.backstage/health/v1/readiness" status=503 httpVersion="1.1" userAgent="kube-probe/1.30" contentLength=58 trace_id="209ea36cf5ed76865373f90cae707c11" span_id="00c9d5899637637b" trace_flags="01" 2025-09-08T12:05:50.179Z backstage info Plugin initializa
- is duplicated by
-
RHDHBUGS-2017 Helm Chart - rhdh-developer-hub pod fails its readiness probe (503)
-
- Closed
-
-
RHDHBUGS-2018 Helm Chart - rhdh-developer-hub pod fails its readiness probe (503)
-
- Closed
-
- relates to
-
FLPATH-2668 Sonataflow di and jobs pods were removed during 1.6->1.7 operator upgrade
-
- Closed
-
- links to