This RFE proposes enhancing the ansible.eda.kafka source plugin within Red Hat Ansible Automation Platform (EDA) to allow a single Kafka source definition to subscribe to and consume messages from multiple Kafka topics. Currently, the topic parameter in the ansible.eda.kafka source only accepts a single string value, meaning users must define beforehand a separate ansible.eda.kafka source block for each individual Kafka topic they wish to monitor within a rulebook.

The requested functionality would involve modifying the topic or introducing a new topics parameter (as initially attempted by the customer) that accepts a list of Kafka topic names. This would streamline rulebook definitions, especially in dynamic environments where the number of Kafka topics to monitor can change frequently or be generated programmatically. 

That would be the case for instance when running a fleet of clusters managed by the Multi Global Hub Operator where its managed clusters will automatically create a new topic to send events.

 

Why is need

 

1. Simplified Rulebook Management for Dynamic Environments: In dynamic, cloud-native, or distributed environments (such as those managed by Red Hat Advanced Cluster Management for Kubernetes - RHACM), new Kafka topics may be created automatically as new clusters or components are onboarded. Manually updating rulebooks to add a new ansible.eda.kafka source block for each new topic is a manual process,  error-prone, and does not scale efficiently.

1. Reduced Operational Overhead: Automating the monitoring of new Kafka topics without manual rulebook modifications significantly reduces the operational burden on administrators and automation engineers. This is particularly relevant when topics are generated based on events (e.g., new OCP cluster joins the hub, creating a dedicated topic).

1. Improved Automation Efficiency: By allowing a single rulebook to centrally listen to a defined set of related topics, the overall automation logic becomes more cohesive and efficient. It avoids the need for potentially duplicate rule logic across multiple single-topic source definitions.

1. Better Alignment with Underlying Library Capabilities: As noted in the customer's internal discussion, the underlying Python Kafka client library (aiokafka) used by the plugin likely supports subscribing to multiple topics. Exposing this capability in the ansible.eda.kafka source would align EDA with the full feature set of its dependencies.

 

Functional Requirements

 

1. Modify the ansible.eda.kafka source plugin: The plugin should be updated to accept a list of strings for a topics parameter (or modify the existing topic parameter to accept a list). The topics parameter should fully support variable expansion, allowing the list of topics to be dynamically sourced from Ansible variables, potentially populated from external sources (e.g., an inventory script, a configuration management database, or a dedicated vars file). This is crucial for enabling the automation use case described above.

Example with variable:

1. In an inventory, vars file, or controller extra_vars

kafka_monitored_topics:

  - topicA

  - topicB

  - topicC

 

1. In the rulebook

• name: Read messages from Kafka topics defined by variable

  hosts: all

  sources:

    - ansible.eda.kafka:

        host: "{{ HOST }}"

        port: 443

        topics: "{{ kafka_monitored_topics }}"

        # ... other parameters

  rules:

    - name: Process Kafka Message

      # ...

 

1. Optional support for wildcard extension. Using a wildcard or regexp expression can easily help to automatically include Kafka topics that are being created when new clusters join the managed fleet of clusters. Also stop monitoring those topics that can be potentially deleted by the Strimzi operator.

             Example with variable:

1. In an inventory, vars file, or controller extra_vars

kafka_monitored_topics:

  - topicA

  - topicB

  - topicC

 

1. In the rulebook

• name: Read messages from Kafka topics defined by variable

  hosts: all

  sources:

    - ansible.eda.kafka:

        host: "{{ HOST }}"

        port: 443

        topic: gh-status.*

        # ... other parameters

  rules:

    - name: Process Kafka Message

      # ...

 

 List any affected known dependencies: Doc, UI, etc.

 

• Documentation: The official Red Hat Ansible Automation Platform documentation, specifically the Ansible Rulebook guide and the ansible.eda.kafka source plugin documentation, would need to be updated to reflect the new topics parameter and its usage.

• Ansible Automation Platform UI / Event-Driven Ansible Controller:

• Any UI elements related to creating or editing rulebooks that involve Kafka sources might need adjustments to support a list input for topics, rather than a single string.

• The schema validation for rulebook definitions within the controller would need to be updated to accept the list format.

• ansible.eda.kafka Collection: The collection itself would require code changes to implement this functionality.

• Testing: Comprehensive testing would be required to ensure the feature works as expected, including scenarios with large numbers of topics, dynamic topic lists, and error handling.