Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-23102

Downstream verification that Autoscaling controller reacts to related resource changes

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • telemetry-operator
    • None

      Goal: 

      • Verify the bugfix is present and working in downstream

      Verification steps:

      NOTE: It's possible these steps will be destructive to the environment.

      The verification will consist of changing multiple resources that the Autoscaling controller should react to and then checking if it really reacted. It'll be very similar for each of the resource. Running `watch -c oc get pods aodh-0` in another terminal might be helpful for seeing the pods getting recreated.

      Also, a tool called "cmctl" will be needed. It's a tool used to manage certificates created by the certmanager. To install see: https://cert-manager.io/docs/reference/cmctl/#manual-installation You may need to modify your PATH based on where and how you install cmctl.

      Before starting the verification itself. Ensure openstack is deployed and telemetry, heat, metric-storage and autoscaling are enabled.

      • Verify the Autoscaling controller reacts to combined-ca-bundle changing (the change can be triggered by renewing the rootca-internal certificate)
        1. Renew the rootca-internal certificate with: `cmctl renew rootca-internal`
        2. Wait until everything settles (could take like a minute, 2 minutes at most)
        3. Verify that the aodh-0 pod was recreated (You'll see that on the age of the pod either in your second terminal if you're using the watch command from above or by executing `oc get pods aodh-0`)
        4. Get the relevant telemetry-operator logs with: `oc -n openstack-operators logs -l app.kubernetes.io/name=telemetry-operator --tail=-1 | | grep Controllers.Autoscaling | grep -v "{"`
        5. Verify that "input source combined-ca-bundle changed, reconcile: autoscaling - openstack" is present among the logs
      • Verify the Autoscaling controller reacts to aodh-internal-svc changes
        1. Renew the aodh-internal-svc certificate with: `cmctl renew aodh-internal-svc`
        2. Verify the aodh-0 pod was recreated (same as above)
        3. Get the relevant telemetry-operator logs (same as above)
        4. Verify that "input source cert-aodh-internal-svc changed, reconcile: autoscaling - openstack" is among the logs
      • Verify the Autoscaling controller reacts to aodh password changes
        1. Change aodh password like this: Execute `oc edit secret osp-secret` change the AodhPassword field contents to "NzY3YzNlZDA1NmNiYWEzYjlkZmVkYjhjNmY4MjViZjA=", exit the editor and save the new contents
        2. Verify the aodh-0 pod was recreated (same as above)
        3. Get the relevant telemetry-operator logs (same as above)
        4. Verify that "input source osp-secret changed, reconcile: autoscaling - openstack" is among the logs
      • Verify that Autoscaling controller reacts to topology changes
        1. Save the following into a file
      apiVersion: topology.openstack.org/v1beta1
      kind: Topology
      metadata:
        name: telemetry-topology
      spec:
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: "topology.kubernetes.io/hostname"
          whenUnsatisfiable: ScheduleAnyway 
        1. Execute `oc apply -f <filename>`
        2. Configure telemetry to use that topology with `oc edit oscp`. Set the `.spec.telemetry.template.topologyRef` to "telemetry-topology"
        3. See the aodh-0 pod restart
        4. Wait until `oc get autoscaling` says autoscaling is ready
        5. Edit the topology (Execute `oc edit topology telemetry-topology`, change the maxSkew to 2)
        6. Verify the aodh-0 pod recreated
        7. Get the relevant telemetry-operator logs (same as above)
        8. Verify that "input source telemetry-topology changed, reconcile: autoscaling - openstack" is among the logs

      Acceptance Criteria:

      • Verification is successful

              Unassigned Unassigned
              rh-ee-jwysogla Jaromir Wysoglad
              rhos-conplat-observability
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: