Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25408

CVO complains "invalid PromQL result length must be one, but is 2" when evaluating PromQL cluster condition: "cluster_installer"

XMLWordPrintable

    • Important
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Patch cv.spec.upstream to use below dummy cincinnati. CVO complains 
      
      65648 I1214 12:48:44.403462       1 promql.go:170] evaluate PromQL cluster condition: "cluster_installer"
      65649 I1214 12:48:44.431854       1 availableupdates.go:134] Requeue available-update evaluation, because "4.15.0-0.nightly-2023-12-17-0      00000" is Recommended=Unknown: EvaluationFailed: Could not evaluate exposure to update risk SomeInvokerThing (invalid PromQL resul      t length must be one, but is 2)
      
      
      {
        "nodes": [
          {
            "version": "4.15.0-0.nightly-2023-12-14-023407",
            "payload": "registry.ci.openshift.org/ocp/release@sha256:a40d2a3e47643a4546063c01ff6c4097162ea31802d3d0ecfa54abbfbc89c884"
          },
          {
            "version": "4.15.0-0.nightly-2023-12-17-000000",
            "payload": "registry.ci.openshift.org/ocp/release@sha256:66c753e8b75d172f2a3f7ba13363383a76ecbc7ecdc00f3a423bef4ea8560405"
          },
          {
            "version": "4.15.0-0.nightly-2023-12-18-111111",
            "payload": "registry.ci.openshift.org/ocp/release@sha256:a5cd1b44e5b25b8a617d92a1f947297f56fc9bad104c117a8e452f932e1e2fd0"
          },
          {
            "version": "4.15.0-0.nightly-2023-12-19-222222",
            "payload": "registry.ci.openshift.org/ocp/release@sha256:e385a786f122c6c0e8848ecb9901f510676438f17af8a5c4c206807a9bc0bf28"
          },
          {
            "version": "4.15.0-0.nightly-2023-12-20-222222",
            "payload": "registry.ci.openshift.org/ocp/release@sha256:caf073ce29232978c331d421c06ca5c2736ce5461962775fdd760b05fb2496a0"
          }
        ],
        "edges": [
          [0,1],
          [0,2],
          [0,3],
          [0,4]
        ],
        "conditionalEdges":[
          {
            "edges": [
              {"from": "4.15.0-0.nightly-2023-12-14-023407", "to": "4.15.0-0.nightly-2023-12-17-000000"}
            ],
            "risks": [
              {
                "url": "https://bug.example.com/a",
                "name": "SomeInvokerThing",
                "message": "On clusters on default invoker user, this imaginary bug can happen.",
                "matchingRules": [
                  {
                    "type": "PromQL",
                    "promql": {
                      "promql": "cluster_installer"
                    }
                  }
                ]
              },
              {
                "url": "https://bug.example.com/b",
                "name": "SomeChannelThing",
                "message": "On clusters with the channel set to 'buggy', this imaginary bug can happen.",
                "matchingRules": [
                  {
                    "type": "PromQL",
                    "promql": {
                      "promql": "group(cluster_version_available_updates{channel=\"buggy\"})\nor\n0 * group(cluster_version_available_updates{channel!=\"buggy\"})"
                    }
                  }
                ]
              }
            ]
          },
          {
            "edges": [
              {"from": "4.15.0-0.nightly-2023-12-14-023407", "to": "4.15.0-0.nightly-2023-12-18-111111"}
            ],
            "risks": [
              {
                "url": "https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.10.0-0.nightly/release/4.10.0-0.nightly-2021-11-24-075634",
                "name": "ReleaseIsRejected",
                "message": "Too many CI failures on this release, so do not update to it",
                "matchingRules": [
                  {
                    "type": "Always"
                  }
                ]
              }
            ]
          },
          {
            "edges": [
              {"from": "4.15.0-0.nightly-2023-12-14-023407", "to": "4.15.0-0.nightly-2023-12-19-222222"}
            ],
            "risks": [
              {
                "url": "https://bug.example.com/c",
                "name": "SomeInfrastructureThing",
                "message": "Clusters on nonexist provider, this imaginary bug can happen.",
                "matchingRules": [
                  {
                    "type": "PromQL",
                    "promql": {
                      "promql": "cluster_infrastructure_provider{type=~\"nonexist\"}\nor\n0 * cluster_infrastructure_provider"
                    }
                  }
                ]
              }
            ]
          },
          {
            "edges": [
              {"from": "4.15.0-0.nightly-2023-12-14-023407", "to": "4.15.0-0.nightly-2023-12-20-222222"}
            ],
            "risks": [
              {
                "url": "https://bug.example.com/d",
                "name": "SomeVersionThing",
                "message": "Fake version, bug https://bug.example.com/d can happen.",
                "matchingRules": [
                  {
                    "type": "PromQL",
                    "promql": {
                      "promql": "0 * group(cluster_version)"
                    }
                  }
                ]
              },
              {
                "url": "https://bug.example.com/e",
                "name": "SomeVersionThing1",
                "message": "Fake version, bug https://bug.example.com/e can happen.",
                "matchingRules": [
                  {
                    "type": "PromQL",
                    "promql": {
                      "promql": "0 * 0 * group(cluster_version)"
                    }
                  }
                ]
              },
              {
                "url": "https://bug.example.com/f",
                "name": "SomeVersionThing2",
                "message": "Fake version, bug https://bug.example.com/f can happen.",
                "matchingRules": [
                  {
                    "type": "PromQL",
                    "promql": {
                      "promql": "0 * 0* 0 * group(cluster_version)"
                    }
                  }
                ]
              }
            ]
          }
        ]
      }    

      Version-Release number of selected component (if applicable):

      4.15.0-0.nightly-2023-12-14-023407    

      How reproducible:

      3/3

      Steps to Reproduce:

          1. Install a 4.15 cluster
          2. Patch cv.spec.upstream to use dummy cincinnati
          3.
          

      Actual results:

      # oc adm upgrade --include-not-recommended
      Cluster version is 4.15.0-0.nightly-2023-12-14-023407
      Upstream: https://raw.githubusercontent.com/shellyyang1989/upgrade-cincy/master/cincy-conditional-edge.json
      Channel: test
      
      Recommended updates:  VERSION                            IMAGE
        4.15.0-0.nightly-2023-12-20-222222 registry.ci.openshift.org/ocp/release@sha256:caf073ce29232978c331d421c06ca5c2736ce5461962775fdd760b05fb2496a0
        4.15.0-0.nightly-2023-12-19-222222 registry.ci.openshift.org/ocp/release@sha256:e385a786f122c6c0e8848ecb9901f510676438f17af8a5c4c206807a9bc0bf28
      
      Supported but not recommended updates:  Version: 4.15.0-0.nightly-2023-12-18-111111
        Image: registry.ci.openshift.org/ocp/release@sha256:a5cd1b44e5b25b8a617d92a1f947297f56fc9bad104c117a8e452f932e1e2fd0
        Recommended: False
        Reason: ReleaseIsRejected
        Message: Too many CI failures on this release, so do not update to it https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.10.0-0.nightly/release/4.10.0-0.nightly-2021-11-24-075634  Version: 4.15.0-0.nightly-2023-12-17-000000
        Image: registry.ci.openshift.org/ocp/release@sha256:66c753e8b75d172f2a3f7ba13363383a76ecbc7ecdc00f3a423bef4ea8560405
        Recommended: Unknown
        Reason: EvaluationFailed
        Message: Could not evaluate exposure to update risk SomeInvokerThing (invalid PromQL result length must be one, but is 2)
          SomeInvokerThing description: On clusters on default invoker user, this imaginary bug can happen.
          SomeInvokerThing URL: https://bug.example.com/a
          

      Expected results:

      CVO is able to evaluate all the risks

      Additional info:

      CVO log is available here: https://drive.google.com/file/d/1NtjD3ktiM-2iDlE0EcAy23VmP3zkhRZ3/view?usp=sharing

              lmohanty@redhat.com Lalatendu Mohanty
              yanyang@redhat.com Yang Yang
              Yang Yang Yang Yang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: