Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4132

File Integrity Operator TargetDown on installs with orignal and new service

XMLWordPrintable

    • ?
    • Moderate
    • 1
    • CMP Sprint 58
    • 1
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Installs from pre-4.8 changes have a file-integrity-operator-metrics service with port 8383 and 8686. This service has an ownerReference to the FIO deployment. The 4.9+ release of FIO creates a new service with ports 8383 and 8585.

      NAME                              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
      file-integrity-operator-metrics   ClusterIP   10.98.75.1   <none>        8383/TCP,8686/TCP   576d
      metrics                           ClusterIP   10.98.34.3   <none>        8383/TCP,8585/TCP   374d
      
          - name: cr-metrics
            port: 8686
            protocol: TCP
            targetPort: 8686

      This creates the corresponding SM

        kind: ServiceMonitor
        metadata:
          creationTimestamp: "2021-04-26T19:12:54Z"
          generation: 1
          labels:
            name: file-integrity-operator
          name: file-integrity-operator-metrics
          namespace: openshift-file-integrity
          ownerReferences:
          - apiVersion: v1
            blockOwnerDeletion: true
            controller: true
            kind: Service
            name: file-integrity-operator-metrics
            uid: 4aa67dae-eb8e-4fd9-bdca-7466301216ce
          resourceVersion: "910050436"
          uid: d545fdf6-08fe-412d-8612-d1de802613f5
        spec:
          endpoints:
          - bearerTokenSecret:
              key: ""
            port: http-metrics
          - bearerTokenSecret:
              key: ""
            port: cr-metrics
          namespaceSelector: {}
          selector:
            matchLabels:
              name: file-integrity-operator 

      You can see the down is on cr-metrics port 8686

              "labels": {
                "endpoint": "cr-metrics",
                "instance": "10.97.5.105:8686",
                "job": "file-integrity-operator-metrics",
                "namespace": "openshift-file-integrity",
                "pod": "file-integrity-operator-69d6bffd64-28q4z",
                "service": "file-integrity-operator-metrics"
              },
              "scrapePool": "serviceMonitor/openshift-file-integrity/file-integrity-operator-metrics/1",
              "scrapeUrl": "http://10.97.5.105:8686/metrics",
              "globalUrl": "http://10.97.5.105:8686/metrics",
              "lastError": "Get \"http://10.97.5.105:8686/metrics\": dial tcp 10.97.5.105:8686: connect: connection refused",
              "lastScrape": "2022-11-24T16:55:42.668150322Z",
              "lastScrapeDuration": 0.000390758,
              "health": "down",
              "scrapeInterval": "30s",
              "scrapeTimeout": "10s" 

       

      How reproducible:

      Always on older installs which have been upgraded.
      

      Steps to Reproduce:

      1. Install a pre 4.8 release which creates the file-integrity-operator-metrics service
      2. Upgrade to the latest release where it creates the metrics service
      3. See target down on port 8686 
      

      Actual results:

      TargrtDown on port 8686 SM

      Expected results:

      No TargetDown alerts. We should cleanup / remove that old service.

      Additional info:

       

       

       

       

            wenshen@redhat.com Vincent Shen
            rhn-support-mrobson Matt Robson
            Xiaojie Yuan Xiaojie Yuan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: