Uploaded image for project: 'Red Hat Developer Hub Bugs'
  1. Red Hat Developer Hub Bugs
  2. RHDHBUGS-2223

[Operator] Breaking change when the Operator is upgraded from 1.7 to 1.8 with an existing 1.7 RHDH CR with Orchestrator: users need to manually update their dynamic-plugins ConfigMap to set the Orchestrator plugins to 1.8.2

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 1.8.0
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide
      = When upgrading the {product-short} Operator to 1.8, users running an existing {product-short} instance with the Orchestrator enabled must manually update their dynamic-plugins ConfigMap to change the version of the Orchestrator plugins from 1.7.1 to 1.8.2

      If you have an existing Operator-backed instance of {product-short} with the Orchestrator, you must update your dynamic-plugins ConfigMap to set the version of the Orchestrator plugins to 1.8.2 once the {product-short} Operator is upgraded to 1.8.
      Otherwise, the {product-short} instance will not be upgraded at all.

      .Example of a dynamic-plugins ConfigMap enabling the Orchestrator plugins in 1.8 for Operator-backed instances
      [source,yaml,subs='+attributes,+quotes']
      ----
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: dynamic-plugins-rhdh
      data:
        dynamic-plugins.yaml: |
          includes:
            - dynamic-plugins.default.yaml
          plugins:
            - package: "@redhat/backstage-plugin-orchestrator@1.8.2"
              disabled: false
            - package: "@redhat/backstage-plugin-orchestrator-backend-dynamic@1.8.2"
              disabled: false
              dependencies:
                - ref: sonataflow
            - package: "@redhat/backstage-plugin-scaffolder-backend-module-orchestrator-dynamic@1.8.2"
              disabled: false
            - package: "@redhat/backstage-plugin-orchestrator-form-widgets@1.8.2"
              disabled: false
      ----
      Show
      = When upgrading the {product-short} Operator to 1.8, users running an existing {product-short} instance with the Orchestrator enabled must manually update their dynamic-plugins ConfigMap to change the version of the Orchestrator plugins from 1.7.1 to 1.8.2 If you have an existing Operator-backed instance of {product-short} with the Orchestrator, you must update your dynamic-plugins ConfigMap to set the version of the Orchestrator plugins to 1.8.2 once the {product-short} Operator is upgraded to 1.8. Otherwise, the {product-short} instance will not be upgraded at all. .Example of a dynamic-plugins ConfigMap enabling the Orchestrator plugins in 1.8 for Operator-backed instances [source,yaml,subs='+attributes,+quotes'] ---- apiVersion: v1 kind: ConfigMap metadata:   name: dynamic-plugins-rhdh data:   dynamic-plugins.yaml: |     includes:       - dynamic-plugins.default.yaml     plugins:       - package: "@redhat/backstage-plugin-orchestrator@1.8.2"         disabled: false       - package: "@redhat/backstage-plugin-orchestrator-backend-dynamic@1.8.2"         disabled: false         dependencies:           - ref: sonataflow       - package: "@redhat/backstage-plugin-scaffolder-backend-module-orchestrator-dynamic@1.8.2"         disabled: false       - package: "@redhat/backstage-plugin-orchestrator-form-widgets@1.8.2"         disabled: false ----
    • Removed Functionality
    • Proposed

      Description of problem:

      Context: https://github.com/redhat-developer/rhdh-operator/pull/1845#pullrequestreview-3424293550

      Prerequisites (if any, like setup, operators/versions):

      • OCP cluster (tested on ROSA 4.19.17)

      Steps to Reproduce

      Working from https://github.com/redhat-developer/rhdh-operator

      1. Switch to the release-1.7 branch
      2. Deploy the Orchestrator plugin infra (prerequisites for using the Orchestrator): `make plugin-infra`
      3. Deploy the 1.7 Operator with `make deploy`
      4. Create the Orchestrator example CR: `oc apply -f examples/orchestrator.yaml`
      5. Wait until CR is reconciled and access the application to confirm that the Orchestrator loads correctly in the RHDH UI:

      6. Now switch to the `release-1.8` branch and perform an upgrade of the Operator using `make deploy`.
      7. Wait until the rhdh-operator pod is replaced and the previous CR is reconciled again

      Actual results:

      No upgrade at all since the new RHDH pod will end up in CrahsLoopBackOff state with an error in the install-dynamic-plugins init container:

      $ oc get pods                                                                                   
      NAME                                                     READY   STATUS                  RESTARTS        AGE
      backstage-orchestrator-6fd67b4f-77kzd                    1/1     Running                 0               16m
      backstage-orchestrator-945b788b6-6rjbl                   0/1     Init:CrashLoopBackOff   6 (2m30s ago)   9m47s
      backstage-psql-orchestrator-0                            1/1     Running                 0               16m
      sonataflow-platform-data-index-service-cf68d6777-jqpff   1/1     Running                 3 (14m ago)     16m
      sonataflow-platform-jobs-service-95f9f468d-lrddq         1/1     Running                 5 (14m ago)     16m
      

      Logs from the init container:

      [...]
      ======= Skipping disabled dynamic plugin @redhat/backstage-plugin-orchestrator@1.8.2
      
      ======= Skipping disabled dynamic plugin @redhat/backstage-plugin-orchestrator-backend-dynamic@1.8.2
      
      ======= Skipping disabled dynamic plugin @redhat/backstage-plugin-scaffolder-backend-module-orchestrator-dynamic@1.8.2
      
      ======= Skipping disabled dynamic plugin @redhat/backstage-plugin-orchestrator-form-widgets@1.8.2
      
      ======= Skipping download of already installed dynamic plugin ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-adoption-insights (already_installed)
              ==> Merging plugin-specific configuration
      
      ======= Skipping download of already installed dynamic plugin ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-adoption-insights-backend-dynamic (already_installed)
              ==> Merging plugin-specific configuration
      
      ======= Skipping download of already installed dynamic plugin ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-analytics-module-adoption-insights-dynamic (already_installed)
              ==> Merging plugin-specific configuration
      
      ======= Installing dynamic plugin @redhat/backstage-plugin-orchestrator@1.7.1
      Traceback (most recent call last):
        File "/opt/app-root/src/install-dynamic-plugins.py", line 658, in <module>
          main()
        File "/opt/app-root/src/install-dynamic-plugins.py", line 644, in main
          _, plugin_config = install_plugin(plugin, plugin_path_by_hash, dynamicPluginsRoot, skipIntegrityCheck)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/src/install-dynamic-plugins.py", line 377, in install_plugin
          plugin_path = installer.install(plugin, plugin_path_by_hash)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/src/install-dynamic-plugins.py", line 273, in install
          raise InstallException(f"No integrity hash provided for Package {package}")
      InstallException: No integrity hash provided for Package @redhat/backstage-plugin-orchestrator@1.7.1
      [...]
      

      The error I reported in https://github.com/redhat-developer/rhdh-operator/pull/1845/files#r2496088934 is different, because I believe the quay.io/rhdh/rhdh-hub-rhel9:1.8 image did not contain the changes from https://github.com/redhat-developer/rhdh/pull/3658/files#diff-2a8ec8751c51963220c18905adc57407d70bbb6647e6fe394cc224291b93a11f yet when I reviewed the PR.

      Expected results:

      The upgrade should be seamless.

      Because of the way a user can enable the Orchestrator using the Operator, they would need to specify the plugin versions in their dynamic-plugins ConfigMap (docs): https://github.com/redhat-developer/rhdh-operator/blob/release-1.7/examples/orchestrator.yaml#L9-L19

      Now, when the Operator is upgraded to 1.8 and the existing RHDH instance is recreated, it will try to load the 1.7.1 plugins (since this is what is configured in the user-provided dynamic-plugins ConfigMap).

      WORKAROUND
      To fix this, the user needs to manually go ahead and change the versions of the Orchestrator plugins to 1.8.2 in their dynamic-plugins ConfigMap, which is a breaking change (from the Install perspective, IMO). The RHDH Operator is supposed to be at the Level 2 - Seamless Upgrades maturity level.

      Reproducibility (Always/Intermittent/Only Once):

      Always.

      Build Details:

      Additional info (Such as Logs, Screenshots, etc):

      Logs attached.

        1. screenshot-1.png
          38 kB
          Armel Soro
        2. install-dynamic-plugins.logs.txt
          12 kB
          Armel Soro

              Unassigned Unassigned
              rh-ee-asoro Armel Soro
              RHIDP - Install
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: