Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-78116

mcd_config_drift metrics is visible even after removing the MCP out of degrade.

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

       For mcd_config_drift showing metric even after resolving the issue

      Version-Release number of selected component (if applicable):

          

      How reproducible:

          

      Steps to Reproduce:

      1. Force the change which will create config drift in node
      $ oc debug node/ip-10-0-117-156.us-west-1.compute.internal -- chroot /host bash -c "echo '# CONFIG DRIFT TEST PR5632' >> /etc/kubernetes/kubelet.conf"
         
      2. Check MCP is degraded
      $ oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-f949a0b4b582fb2000326c35f0b2cebc   True      False      False      3              3                   3                     0                      146m
      worker   rendered-worker-97364c44c883a6f8dc726d2053c88d2a   False     True       True       3              2                   2                     1                      146m     
      3. Check config drift is produced oc exec -n openshift-machine-config-operator machine-config-daemon-jc2cm  -- \
          curl -s localhost:8797/metrics 2>&1 | grep mcd_config_drift
      # HELP mcd_config_drift timestamp for config drift
      # TYPE mcd_config_drift gauge
      mcd_config_drift 1.7731343462747185e+09
      
      4. Revert the changes
      $ oc debug node/ip-10-0-117-156.us-west-1.compute.internal --    chroot /host bash -c "sed -i '/# CONFIG DRIFT TEST PR5632/d' /etc/kubernetes/kubelet.conf"
      
      5. Check MCP is no more degrade
      $ oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-f949a0b4b582fb2000326c35f0b2cebc   True      False      False      3              3                   3                     0                      149m
      worker   rendered-worker-97364c44c883a6f8dc726d2053c88d2a   True      False      False      3              3                   3                     0                      149m
      
      6. Able to still see the mcd_config_drift showing the metrics
      $ oc exec -n openshift-machine-config-operator machine-config-daemon-jc2cm  -- \
          curl -s localhost:8797/metrics 2>&1 | grep mcd_config_drift
      # HELP mcd_config_drift timestamp for config drift
      # TYPE mcd_config_drift gauge
      mcd_config_drift 1.7731344167761781e+09

      Actual results:

       

      Expected results:

           mcd_metric_config should be showing zero config drift after resolving 
      $ oc exec -n openshift-machine-config-operator machine-config-daemon-jc2cm  -- \     curl -s localhost:8797/metrics 2>&1 | grep mcd_config_drift # HELP mcd_config_drift timestamp for config drift # TYPE mcd_config_drift gauge mcd_config_drift 0

      Additional info:

          

              team-mco Team MCO
              rh-ee-ptalgulk Prachiti Talgulkar
              Sergio Regidor de la Rosa Sergio Regidor de la Rosa
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: