• Icon: Spike Spike
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • None

      This spike is to try to identify ways to ingest OpenStack cluster-level data to Insights.

      I connected with the Insights and CCX team and they gave me some guidance.

      A potential way to accomplish this could be to add OSP CRD(s) to the OIO (OpenShift Insights Operator) gatherers. A PR can be sent and we can use "Cluster-bot" Slack App to spin up an OCP cluster that gathers data from our CRDs.

      Things to investigate as part of this spike:

      • Can we use the OSP meta operator to collect all the info we want to expose and have this gathered by the OIO? Or any other alternative (vs. multiple gatherers, or a dedicated operator for OSP Insights)
      • Any potential limits (the Insights team mentioned, that they collect the data every 2 hours and there's a limit of 8MB per archive)

      I'd suggest having a small PoC that can feed basic info about an OSP cluster such as the topology ([eg|https://github.com/openshift/insights-operator/tree/master/docs/insights-archive-sample/config/node]. of OCP cluster topology). As a bonus, we can try to query this data via Trino (eg. what's the max number of nodes of an OSP deployment).

      For us to verify if the data is gathered we can ping the Insights team (contact tremes1@redhat.com) with the cluster ID (created by cluster-bot) and they should be able to give us more information.

      To consider this Spike as done we should have written document https://docs.google.com/document/d/1r3sC_7ZU7qkxvafpEkAJKMTmtcWAwGOI6W_SZGkvP6s/edit with info about:

      • list of options to implement, with pros and cons enumerated
      • decision about preferable path to go with (after discussing various options with the Insights team representative,
      • PoC showing that choosen solution works and we can move on with it.

            [OSPRH-4003] Evaluate how to ingest data into Insights

            Daniel Alvarez Sanchez created issue -
            Ihar Hrachyshka made changes -
            Epic Link New: OSP-27289 [ 15412228 ]
            Ihar Hrachyshka made changes -
            Assignee New: Slawomir Kaplonski [ skaplons@redhat.com ]
            Slawomir Kaplonski made changes -
            Key Original: OSP-27291 New: OSPRH-4003
            PM Score Original: 0
            Regression New: No [ 17372 ]
            Workflow Original: Bugzilla Mirror - Simple Backlog [ 24071146 ] New: OJA-WF-K [ 24932729 ]
            Project Original: OpenStack Platform [ 12326520 ] New: Red Hat OpenStack Services on OpenShift [ 12336920 ]
            Slawomir Kaplonski made changes -
            Workstream New: Networking; Neutron [ 28866 ]
            Slawomir Kaplonski made changes -
            Description Original: This spike is to try to identify ways to ingest OpenStack cluster-level data to Insights.

            I connected with the Insights and CCX team and they gave me some guidance.

            A potential way to accomplish this could be to add OSP CRD(s) to the OIO (OpenShift Insights Operator) [gatherers|https://github.com/openshift/insights-operator/blob/37007330f6169f78f823fc6a95f30f882897cf91/pkg/gatherers/clusterconfig/gather_custom_resource_definitions.go#L51-L58]. A PR can be sent and we can use "Cluster-bot" Slack App to spin up an OCP cluster that gathers data from our CRDs.

            Things to investigate as part of this spike:
             * Can we use the OSP meta operator to collect all the info we want to expose and have this gathered by the OIO? Or any other alternative (vs. multiple gatherers, or a dedicated operator for OSP Insights)
             * Any potential limits (the Insights team mentioned, that they collect the data every 2 hours and there's a limit of 8MB per archive)

            I'd suggest having a small PoC that can feed basic info about an OSP cluster such as the topology ([eg|[https://github.com/openshift/insights-operator/tree/master/docs/insights-archive-sample/config/node]]. of OCP cluster topology). As a bonus, we can try to query this data via Trino (eg. what's the max number of nodes of an OSP deployment).

            For us to verify if the data is gathered we can ping the Insights team (contact [~tremes1@redhat.com]) with the cluster ID (created by cluster-bot) and they should be able to give us more information.
            New: This spike is to try to identify ways to ingest OpenStack cluster-level data to Insights.

            I connected with the Insights and CCX team and they gave me some guidance.

            A potential way to accomplish this could be to add OSP CRD(s) to the OIO (OpenShift Insights Operator) [gatherers|https://github.com/openshift/insights-operator/blob/37007330f6169f78f823fc6a95f30f882897cf91/pkg/gatherers/clusterconfig/gather_custom_resource_definitions.go#L51-L58]. A PR can be sent and we can use "Cluster-bot" Slack App to spin up an OCP cluster that gathers data from our CRDs.

            Things to investigate as part of this spike:
             * Can we use the OSP meta operator to collect all the info we want to expose and have this gathered by the OIO? Or any other alternative (vs. multiple gatherers, or a dedicated operator for OSP Insights)
             * Any potential limits (the Insights team mentioned, that they collect the data every 2 hours and there's a limit of 8MB per archive)

            I'd suggest having a small PoC that can feed basic info about an OSP cluster such as the topology ([eg|[https://github.com/openshift/insights-operator/tree/master/docs/insights-archive-sample/config/node]]. of OCP cluster topology). As a bonus, we can try to query this data via Trino (eg. what's the max number of nodes of an OSP deployment).

            For us to verify if the data is gathered we can ping the Insights team (contact [~tremes1@redhat.com]) with the cluster ID (created by cluster-bot) and they should be able to give us more information.

            To consider this Spike as done we should have written document [https://docs.google.com/document/d/1r3sC_7ZU7qkxvafpEkAJKMTmtcWAwGOI6W_SZGkvP6s/edit] with info about:
             * list of options to implement, with pros and cons enumerated
             * decision about preferable path to go with (after discussing various options with the Insights team representative,
             * PoC showing that choosen solution works and we can move on with it.
            Jesse Pretorius made changes -
            Team New: rhos-dfg-networking-squad-neutron
            Slawomir Kaplonski made changes -
            Status Original: New [ 10016 ] New: In Progress [ 10018 ]
            Slawomir Kaplonski made changes -
            Status Original: In Progress [ 10018 ] New: Review [ 12422 ]
            Slawomir Kaplonski made changes -
            Link New: This issue is triggering OSPRH-5905 [ OSPRH-5905 ]
            Slawomir Kaplonski made changes -
            Link New: This issue is triggering OSPRH-5904 [ OSPRH-5904 ]
            Slawomir Kaplonski made changes -
            Resolution New: Done [ 1 ]
            Status Original: Review [ 12422 ] New: Closed [ 6 ]
            Clark Everson made changes -
            Workflow Original: OJA-WF-K [ 24932729 ] New: OJA-WF-AG [ 25251754 ]
            Slawomir Kaplonski made changes -
            Link New: This issue is triggering OSPRH-9214 [ OSPRH-9214 ]
            Amol Dongare made changes -
            Workflow Original: OJA-WF-AG [ 25251754 ] New: OJA-WF-D [ 26779130 ]

              skaplons@redhat.com Slawomir Kaplonski
              dalvarez@redhat.com Daniel Alvarez Sanchez
              rhos-dfg-networking-squad-neutron
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: