Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-6029

Drainer search in every StatefulSet of the cluster for AMQ broker dependencies

    XMLWordPrintable

Details

    • False
    • False
    • Hide

      Thanks @Mikhail Krutov to for the reproducer:

       

      The way to reproduce it is to have such configuration:

      namespace openshift-operators:

      — amq-broker-operator namespace

      some-other-namespace:

      — statefulset unrelatedStatefulset
      — statefulset oneMoreUnrelatedStatefulset
      — statefulset brokerStatefulset ----- this statefulset has message migration and persistence enabled

      In this case, operator's scaledown controller indeed checks on each statefulset inside of some-other-namespace, but it only operates on brokerStatefulSet, ie. its involvement in unrelated statefulsets is only read-only.

      Show
      Thanks @Mikhail Krutov to for the reproducer:   The way to reproduce it is to have such configuration: namespace openshift-operators: — amq-broker-operator namespace some-other-namespace: — statefulset unrelatedStatefulset — statefulset oneMoreUnrelatedStatefulset — statefulset brokerStatefulset ----- this statefulset has message migration and persistence enabled In this case, operator's scaledown controller indeed checks on each statefulset inside of some-other-namespace, but it only operates on brokerStatefulSet, ie. its involvement in unrelated statefulsets is only read-only.

    Description

       

      When deploying the operator and a new set of brokers is created with persistence and message migration enabled, the operator searches statefulsets across the entire cluster, to find the one related with the brokers.

       

      2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "openshift-user-workload-monitoring/thanos-ruler-user-workload", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "open-cluster-management/search-redisgraph", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "openshift-storage/noobaa-db-pg", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "openshift-user-workload-monitoring/prometheus-user-workload", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "openshift-monitoring/alertmanager-main", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "openshift-monitoring/prometheus-k8s", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "openshift-storage/noobaa-core", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "amqbroker/demo-ss", "controller ns": "*"}
2022-01-18T10:26:12.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Enquequing statefulset	{"key": "hive/hive-clustersync", "controller ns": "*"}
2022-01-18T10:26:17.702Z	INFO	controller_v2alpha1activemqartemisscaledown	Got object from queue	{"obj": "openshift-user-workload-monitoring/thanos-ruler-user-workload"}

       

       

      It seam it is not impacting the drainer as it only performs the logic when the STS comes from a broker, but it has some penalties:

      • Performance: clusters with high number of STS could return a lot members to search within.
      • API calls: it can cause a lot of api calls, as it has to get complimentary information from some STSs that should are not needed (like PVCs).

      Attachments

        Activity

          People

            dbruscin Domenico Francesco Bruscino
            rgordill1@redhat.com Ramon Gordillo Gutierrez
            Mikhail Krutov Mikhail Krutov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: