Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2775

After added/removed label from a namespace, one stats of "route_metrics_controller_routes_per_shard" in Observe >> Metrics page aren't correct

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • 4.12.0
    • 4.12
    • Networking / router
    • None
    • Moderate
    • None
    • Sprint 226, Sprint 227
    • 2
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • NA

      Description of problem:

      There were 4 ingress-controllers and totally 15 routes. On web console, try to query "route_metrics_controller_routes_per_shard" in Observe >> Metrics page. the stats for 3 ingress-controllers are 15, and it is 1 for the last ingress-controller

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-10-23-154914

      How reproducible:

      Create pods, services, ingress-controllers, routes, then check  "route_metrics_controller_routes_per_shard" on web console

      Steps to Reproduce:

      1. get cluster's base domain
      % oc get dnses.config/cluster -oyaml | grep -i domain
        baseDomain: shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      
      2. create 3 clusters
      % oc -n openshift-ingress-operator get ingresscontroller
      NAME         AGE
      default      7h5m
      extertest3   120m
      internal1    120m
      internal2    120m
      % 
      
      3. check the spec of the 4 ingress-controllres
      a, default
      
      b, extertest3
      spec:
        domain: extertest3.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
        endpointPublishingStrategy:
          loadBalancer:
            dnsManagementPolicy: Managed
            scope: External
          type: LoadBalancerService
      c, internal1
      spec:
        domain: internal1.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
        endpointPublishingStrategy:
          loadBalancer:
            dnsManagementPolicy: Managed
            scope: Internal
          type: LoadBalancerService
      d, internal2
      spec:
        domain: internal2.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
        endpointPublishingStrategy:
          loadBalancer:
            dnsManagementPolicy: Managed
            scope: Internal
          type: LoadBalancerService
        routeSelector:
          matchLabels:
            shard: alpha
      
      4. check the route, there are 15 routes
      % oc get route -A | awk '{print $3}'
      HOST/PORT
      oauth-openshift.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      console-openshift-console.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      downloads-openshift-console.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      canary-openshift-ingress-canary.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      alertmanager-main-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      prometheus-k8s-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      prometheus-k8s-federate-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      thanos-querier-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      edge1-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      int1reen2-test.internal1.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      pass1-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      reen1-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      service-unsecure-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      int1edge2-test.internal1.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      test.shudi.com
      %
      
      % oc get route -A | awk '{print $3}' | grep apps.shudi
      oauth-openshift.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      console-openshift-console.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      downloads-openshift-console.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      canary-openshift-ingress-canary.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      alertmanager-main-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      prometheus-k8s-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      prometheus-k8s-federate-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      thanos-querier-openshift-monitoring.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      edge1-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      pass1-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      reen1-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      service-unsecure-test.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com
      %
      
      % oc get route -A | awk '{print $3}' | grep apps.shudi | wc -l
            12
      % oc get route -A | awk '{print $3}' | grep internal1 | wc -l 
             2
      % oc get route -A | awk '{print $3}' | grep shudi.com | wc -l
             1
      %
      
      5. only route unsvc5 had the shard=alpha label
       % oc get route unsvc5  -oyaml | grep labels: -A2
        labels:
          name: unsvc5
          shard: alpha
       % oc get route unsvc5 -oyaml | grep spec: -A1
        spec:
          host: test.shudi.com
      
      6. login web console(https://https://console-openshift-console.apps.shudi-412gcpop36.qe.gcp.devcluster.openshift.com/monitoring/query-browser), then navigate to Observe >> Metrics 
      
      7. input"route_metrics_controller_routes_per_shard ", then click the "Run queries" button. As the attached picture showed:
      ​​name                           value
      default                        15
      extertest3                     15
      internal1                      15      
      internal2                      1
      
      8. Also there was a minor issue: As the attached picture showed, there were two name in the header line
      
      Name                                           name      value                              
      route_metrics_controller_routes_per_shard     default    15
      route_metrics_controller_routes_per_shard     extertest3 15
      route_metrics_controller_routes_per_shard     internal1  15
      route_metrics_controller_routes_per_shard     internal2  1

      Actual results:

      ​​name                         value 
      default                      15
      extertest3                   15 
      internal1                    15
      internal2                    1

      Expected results:

      ​​name                         value
      default                      12
      extertest3                   0
      internal1                    2 
      internal2                    1

      Additional info:

       

              rh-ee-arsen Arkadeep Sen
              shudili@redhat.com Shudi Li
              Shudi Li Shudi Li
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: