Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-188

Database user name change not propogated after update/apply

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • rhos-18.0.0
    • None
    • nova-operator
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      Deploy multi-cell nova environment using https://github.com/openstack-k8s-operators/nova-operator/blob/master/config/samples/nova_v1beta1_nova-multi-cell.yaml

      1. In devsetup
        CPUS=16 MEMORY=18016 make crc
        eval $(crc oc-env)
        make download_tools
        oc login -u kubeadmin -p 12345678 https://api.crc.testing:6443
      1. Install Operators
        cd ~/install_yamls/
        make crc_storage
        make input
        make mariadb
        make keystone
        make placement
        make rabbitmq
        make infra
        make ansibleee
        make nova
      2. May need to wait on OLM setup before doing below commands (make now has sleep)
        make mariadb_deploy
        make keystone_deploy
        make placement_deploy
        make rabbitmq_deploy
      1. Setup OLM for Nova
      1. Setup MariaDB and RabbitMQ for cell1
        mkdir -p ~/install_yamls/out/tmp/mariadb/cr ~/install_yamls/out/tmp/rabbitmq/cr ~/install_yamls/out/tmp/nova/cr
        cd ~/install_yamls/out/tmp/mariadb/cr
        cp ../../../operator/mariadb-operator/config/samples/mariadb_v1beta1_mariadb.yaml mariadb_v1beta1_mariadb_cell1.yaml
      2. Update yaml cr name to -> name: mariadb-cell1
      3. Either copy over or create kustomization.yaml
        cat kustomization.yaml 
        apiVersion: kustomize.config.k8s.io/v1beta1
        kind: Kustomization
        resources:
        namespace: openstack
        patches:
      • patch: |-
            - op: replace
              path: /spec/secret
              value: osp-secret
            - op: replace
              path: /spec/storageClass
              value: "local-storage"
          target:
            kind: MariaDB
        kustomize edit add resource mariadb_v1beta1_mariadb_cell1.yaml
        oc kustomize . | oc apply -f -

      cd ~/install_yamls/out/tmp/rabbitmq/cr
      cp ~/install_yamls/out/openstack/rabbitmq/cr/kustomization.yaml .
      cp ~/install_yamls/out/operator/rabbitmq-operator/docs/examples/default-security-context/rabbitmq.yaml rabbitmq_cell1.yaml

      1. No need to update rabbitmq_cell1.yaml the kustomization.yaml patches name so just update kustomization.yaml
        [stack@compute-sriov01 cr]$ cat kustomization.yaml 
        apiVersion: kustomize.config.k8s.io/v1beta1
        kind: Kustomization
        resources:
      • rabbitmq_cell1.yaml
        namespace: openstack
        patches:
      • patch: |-
            - op: replace
              path: /metadata/name
              value: rabbitmq-cell1
          target:
            kind: RabbitmqCluster

      oc kustomize .
      oc kustomize . | oc apply -f -

      1. Start up Nova CR's
        cd ~/install_yamls/out/tmp/nova/cr/
        cp ~/nova-operator/config/samples/nova_v1beta1_nova* .
      1. Remove cell2 section from nova_v1beta1_nova-multi-cell.yaml 
        vi nova_v1beta1_nova-multi-cell.yaml 
        [stack@compute-sriov01 cr]$ cat kustomization.yaml 
        apiVersion: kustomize.config.k8s.io/v1beta1
        kind: Kustomization
        resources:
      • ./nova_v1beta1_nova-multi-cell.yaml
        namespace: openstack
        patches:
      • patch: |-
            - op: replace
              path: /spec/secret
              value: osp-secret
            - op: replace
              path: /spec/storageClass
              value: "local-storage"
          target:
            kind: Nova
      1. Check kustomize
        oc kustomize .
      1. Apply
        [stack@compute-sriov01 cr]$ oc kustomize . | oc apply -f -

      After deploying change the database username to nova-cell0 and reapply.

      Show
      Deploy multi-cell nova environment using https://github.com/openstack-k8s-operators/nova-operator/blob/master/config/samples/nova_v1beta1_nova-multi-cell.yaml In devsetup CPUS=16 MEMORY=18016 make crc eval $(crc oc-env) make download_tools oc login -u kubeadmin -p 12345678 https://api.crc.testing:6443 Install Operators cd ~/install_yamls/ make crc_storage make input make mariadb make keystone make placement make rabbitmq make infra make ansibleee make nova May need to wait on OLM setup before doing below commands (make now has sleep) make mariadb_deploy make keystone_deploy make placement_deploy make rabbitmq_deploy Setup OLM for Nova Setup MariaDB and RabbitMQ for cell1 mkdir -p ~/install_yamls/out/tmp/mariadb/cr ~/install_yamls/out/tmp/rabbitmq/cr ~/install_yamls/out/tmp/nova/cr cd ~/install_yamls/out/tmp/mariadb/cr cp ../../../operator/mariadb-operator/config/samples/mariadb_v1beta1_mariadb.yaml mariadb_v1beta1_mariadb_cell1.yaml Update yaml cr name to -> name: mariadb-cell1 Either copy over or create kustomization.yaml cat kustomization.yaml  apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: namespace: openstack patches: patch: |-     - op: replace       path: /spec/secret       value: osp-secret     - op: replace       path: /spec/storageClass       value: "local-storage"   target:     kind: MariaDB kustomize edit add resource mariadb_v1beta1_mariadb_cell1.yaml oc kustomize . | oc apply -f - cd ~/install_yamls/out/tmp/rabbitmq/cr cp ~/install_yamls/out/openstack/rabbitmq/cr/kustomization.yaml . cp ~/install_yamls/out/operator/rabbitmq-operator/docs/examples/default-security-context/rabbitmq.yaml rabbitmq_cell1.yaml No need to update rabbitmq_cell1.yaml the kustomization.yaml patches name so just update kustomization.yaml [stack@compute-sriov01 cr] $ cat kustomization.yaml  apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: rabbitmq_cell1.yaml namespace: openstack patches: patch: |-     - op: replace       path: /metadata/name       value: rabbitmq-cell1   target:     kind: RabbitmqCluster oc kustomize . oc kustomize . | oc apply -f - Start up Nova CR's cd ~/install_yamls/out/tmp/nova/cr/ cp ~/nova-operator/config/samples/nova_v1beta1_nova* . Remove cell2 section from nova_v1beta1_nova-multi-cell.yaml  vi nova_v1beta1_nova-multi-cell.yaml  [stack@compute-sriov01 cr] $ cat kustomization.yaml  apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: ./nova_v1beta1_nova-multi-cell.yaml namespace: openstack patches: patch: |-     - op: replace       path: /spec/secret       value: osp-secret     - op: replace       path: /spec/storageClass       value: "local-storage"   target:     kind: Nova Check kustomize oc kustomize . Apply [stack@compute-sriov01 cr] $ oc kustomize . | oc apply -f - After deploying change the database username to nova-cell0 and reapply.
    • Compute

      Incorrect database user was applied in crc environment.  Updating the user and applying the changes are not propagated to nova services.

      [stack@compute-sriov01 cr]$ oc logs nova-scheduler-0 | tail -20
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/_init_.py", line 94, in Connect
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service     return Connection(*args, **kwargs)
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 327, in _init_
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service     self.connect()
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 588, in connect
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service     self._request_authentication()
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 853, in _request_authentication
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service     auth_packet = self._read_packet()
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service     packet.raise_for_error()
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service     err.raise_mysql_exception(self._data)
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service     raise errorclass(errno, errval)
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'nova_cell0'@'10.217.0.144' (using password: YES)")
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service (Background on this error at: https://sqlalche.me/e/14/e3q8)
      2023-05-02 18:41:03.400 1 ERROR oslo_service.service 
      2023-05-02 18:41:03.414 1 DEBUG oslo_concurrency.lockutils [None req-a4d84eba-1ae9-495a-a902-8b1f2ac32ec7 - - - - - -] Acquiring lock "singleton_lock" lock /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:312
      2023-05-02 18:41:03.414 1 DEBUG oslo_concurrency.lockutils [None req-a4d84eba-1ae9-495a-a902-8b1f2ac32ec7 - - - - - -] Acquired lock "singleton_lock" lock /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:315
      2023-05-02 18:41:03.415 1 DEBUG oslo_concurrency.lockutils [None req-a4d84eba-1ae9-495a-a902-8b1f2ac32ec7 - - - - - -] Releasing lock "singleton_lock" lock /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:333
      [stack@compute-sriov01 cr]$ date
      Tue May  2 18:44:49 UTC 2023
      [stack@compute-sriov01 cr]$ grep cellDatabaseUser *
      nova_v1beta1_nova-multi-cell.yaml:      cellDatabaseUser: nova-cell0 
      nova_v1beta1_nova-multi-cell.yaml:      cellDatabaseUser: nova_cell1
      [stack@compute-sriov01 cr]$ oc kustomize . | oc apply -f -
      nova.nova.openstack.org/nova configured

      [stack@compute-sriov01 cr]$ grep cellDatabaseUser *
      nova_v1beta1_nova-multi-cell.yaml:      cellDatabaseUser: nova-cell0 
      nova_v1beta1_nova-multi-cell.yaml:      cellDatabaseUser: nova_cell1
      [stack@compute-sriov01 cr]$ oc kustomize . | oc apply -f -
      nova.nova.openstack.org/nova configured
      [stack@compute-sriov01 cr]$ oc logs nova-scheduler-0 | tail -20
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/_init_.py", line 94, in Connect
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service     return Connection(*args, **kwargs)
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 327, in _init_
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service     self.connect()
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 588, in connect
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service     self._request_authentication()
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 853, in _request_authentication
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service     auth_packet = self._read_packet()
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service     packet.raise_for_error()
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service     err.raise_mysql_exception(self._data)
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service     raise errorclass(errno, errval)
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'nova_cell0'@'10.217.0.144' (using password: YES)")
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service (Background on this error at: https://sqlalche.me/e/14/e3q8)
      2023-05-02 18:46:09.239 1 ERROR oslo_service.service 
      2023-05-02 18:46:09.246 1 DEBUG oslo_concurrency.lockutils [None req-634483d9-4201-4b3e-84ce-9275bc654054 - - - - - -] Acquiring lock "singleton_lock" lock /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:312
      2023-05-02 18:46:09.246 1 DEBUG oslo_concurrency.lockutils [None req-634483d9-4201-4b3e-84ce-9275bc654054 - - - - - -] Acquired lock "singleton_lock" lock /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:315
      2023-05-02 18:46:09.246 1 DEBUG oslo_concurrency.lockutils [None req-634483d9-4201-4b3e-84ce-9275bc654054 - - - - - -] Releasing lock "singleton_lock" lock /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:333

            rh-ee-bgibizer Balazs Gibizer
            rhn-gps-jparker James Parker
            rhos-dfg-compute
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: