Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-45047

Gather LokiStack resources to enable an Advisor recommendation

XMLWordPrintable

      Description of problem:

          Context
OpenShift Logging is migrating from Elasticsearch to Loki. While the option to use Loki has existed forquite a while, the information about end of Elasticsearch support has not been available until recently. With the information available now, we can expect more and more customers to migrate and hit the issue described in INSIGHTOCP-1927.
P.S. Note the bar chart in INSIGHTOCP-1927 which shows how frequently is the related KCS linked in customer cases.
Data to gather
LokiStack custom resources (any name, any namespace)
Backports
The option to use Loki is available since Logging 5.5 whose compatibility started at OCP 4.9. Considering the OCP life cycle, backports to up to OCP 4.14 would be nice.
Unknowns
Since Logging 5.7, Logging supports installation of multiple instances in customer namespaces. The Insights Operator would have to look for the CRs in all namespaces, which poses the following questions:

What is the expected number of the LokiStack CRs in a cluster?
Should the Insights operator look for the resource in all namespaces? Is there a way to narrow down the scope?

The CR will contain the name of a customer namespaces which is a sensitive information.
What is the API group of the CR? Is there a risk of LokiStack CRs in customer namespaces that would NOT be related to OpenShift Logging?



SME
Oscar Arribas Arribas

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

      N/A

      Actual results:

      Expected results:

      Additional info:

            [OCPBUGS-45047] Gather LokiStack resources to enable an Advisor recommendation

            baiyang zhou added a comment - - edited

            Verification version

            4.19.0-0.nightly-2024-12-06-101930

            Verification Type

            Automatic-manual

            Preparation

            Using flexy-install launches an openshift cluster whose image version is 4.19.0-0.nightly-2024-12-06-101930

            Verification steps

            1. Install Loki Operator and Red Hat OpenShift Logging via web console.
            2. Execute the following sh script to get cloud platform certificates.

             

            #!/usr/bin/env bash
#https://grafana.com/docs/loki/latest/storage/
#https://stackoverflow.com/questions/65232062/can-i-setup-a-single-store-on-gcs-and-have-multiple-local-instances-of-loki-read
#https://docs.openshift.com/container-platform/4.10/registry/configuring_registry_storage/configuring-registry-storage-gcp-user-infrastructure.html
SECRETNAME="gcs-secret"
BUCKET_NAME="${1:-loki-aosqe}"
GCS_KEYFILE="${2:-REGISTRY_STORAGE_GCS_KEYFILE}"set_gcp_credentials()
{
  echo "CreateCCPSecret: get GCP credentials"
  #use credentials in Env as the first option
  #use cloud provider as the second option
  if [[ ! -z $GCS_KEYFILE ]]; then
      echo "reuse registry key"
      oc extract secret/image-registry-private-configuration -n openshift-image-registry --confirm
  fi
}create_secret() {
  set_gcp_credentials
  oc delete secret ${SECRETNAME} || :
  oc create secret generic ${SECRETNAME} \
    --from-literal=bucketname="${BUCKET_NAME}" \
    --from-file="key.json"="${GCS_KEYFILE}"
  if [[ $? == 0 ]]  ;then
      echo "CreateCCPSecret: Secret created, please make sure the bucket $BUCKET_NAME exist on GCP"
  fi
}main() {
  create_secret
}main

            3. Using the following configuration to deploy an instance of LokiStack.

             

             

            apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small
  storage:
    schemas:
    - version: v12
      effectiveDate: "2022-06-01"
    secret:
      name: gcs-secret
      type: s3
  storageClassName: gp2
  tenants:
    mode: openshift-logging

            4. Change the `namespace` field to `kube-public` in the above configuration, then deploy it.

             

            5. Using the following codes to create more than 20 instances of LokiStack in openshift-logging namespace.

            loki_resource = cluster.client.resources.get(api_version="loki.grafana.com/v1", kind="LokiStack")
for i in range(30):
    resource = {
        "apiVersion": "loki.grafana.com/v1",
        "kind": "LokiStack",
        "metadata": {
            "name": f"loki-instance-{i}",
            "namespace": "openshift-logging"
        },
        "spec": {
            "size": "1x.small",
            "storage": {
                "schemas": [
                    {
                        "version": "v12",
                        "effectiveDate": "2022-06-01"
                    }
                ],
                "secret": {
                    "name": "gcs-secret",
                    "type": "gcs"
                }
            },
            "storageClassName": "gp2",
            "tenants": {
                "mode": "openshift-logging"
            }
        }
    }
    loki_resource.create(resource)
cluster.restart_insights_operator()

            6. Restart IO

            7. View the Insights archive that path is `insights-operator/gather.json`, check whether the `clusterconfig/lokistack` filed contains the error info about `found resource in an unexpected namespace` and `found 22 resources, limit (20) reached`

            8. View the relevant archives to confirm whether their contents contain the corresponding info of lokiStack resources.

            Expected results

            1.Both of these 2 operators are installed successfully.

            7. The `clusterconfig/lokistack` filed of `gather.json` archive contains the error info about `found resource in an unexpected namespace` and `found 22 resources, limit (20) reached`

            8. The relevant insights archives contain the corresponding info of lokiStack resources.

            Actual results

            1.Both of these 2 operators are installed successfully.

            7. The `clusterconfig/lokistack` filed of `gather.json` archive contains the error info about `found resource in an unexpected namespace` and `found 22 resources, limit (20) reached`

            8. The relevant insights archives contain the corresponding info of lokiStack resources.

            Remark

            Reference for installing operator: https://docs.redhat.com/en/documentation/openshift_container_platform/4.10/html/logging/cluster-logging-loki#logging-loki-deploy_cluster-logging-loki

            Reference for getting cloud platform certificates: https://gitlab.cee.redhat.com/aosqe/aosqe-tools/-/tree/master/logging/log_template/loki-operator?ref_type=heads

            baiyang zhou added a comment - - edited Verification version 4.19.0-0.nightly-2024-12-06-101930 Verification Type Automatic-manual Preparation Using flexy-install launches an openshift cluster whose image version is 4.19.0-0.nightly-2024-12-06-101930 Verification steps Install Loki Operator and Red Hat OpenShift Logging via web console. Execute the following sh script to get cloud platform certificates.   #!/usr/bin/env bash #https: //grafana.com/docs/loki/latest/storage/ #https: //stackoverflow.com/questions/65232062/can-i-setup-a-single-store-on-gcs-and-have-multiple-local-instances-of-loki-read #https: //docs.openshift.com/container-platform/4.10/registry/configuring_registry_storage/configuring-registry-storage-gcp-user-infrastructure.html SECRETNAME= "gcs-secret" BUCKET_NAME= "${1:-loki-aosqe}" GCS_KEYFILE= "${2:-REGISTRY_STORAGE_GCS_KEYFILE}" set_gcp_credentials() {   echo "CreateCCPSecret: get GCP credentials"   #use credentials in Env as the first option   #use cloud provider as the second option   if [[ ! -z $GCS_KEYFILE ]]; then       echo "reuse registry key"       oc extract secret/image-registry- private -configuration -n openshift-image-registry --confirm   fi }create_secret() {   set_gcp_credentials   oc delete secret ${SECRETNAME} || :   oc create secret generic ${SECRETNAME} \     --from-literal=bucketname= "${BUCKET_NAME}" \     --from-file= "key.json" = "${GCS_KEYFILE}"   if [[ $? == 0 ]]  ;then       echo "CreateCCPSecret: Secret created, please make sure the bucket $BUCKET_NAME exist on GCP"   fi }main() {   create_secret }main 3. Using the following configuration to deploy an instance of LokiStack.     apiVersion: loki.grafana.com/v1 kind: LokiStack metadata:   name: logging-loki   namespace: openshift-logging spec:   size: 1x.small   storage:     schemas:     - version: v12       effectiveDate: "2022-06-01"     secret:       name: gcs-secret       type: s3   storageClassName: gp2   tenants:     mode: openshift-logging 4. Change the `namespace` field to `kube-public` in the above configuration, then deploy it.   5. Using the following codes to create more than 20 instances of LokiStack in openshift-logging namespace. loki_resource = cluster.client.resources.get(api_version= "loki.grafana.com/v1" , kind= "LokiStack" ) for i in range(30): resource = { "apiVersion" : "loki.grafana.com/v1" , "kind" : "LokiStack" , "metadata" : { "name" : f "loki-instance-{i}" , "namespace" : "openshift-logging" }, "spec" : { "size" : "1x.small" , "storage" : { "schemas" : [ { "version" : "v12" , "effectiveDate" : "2022-06-01" } ], "secret" : { "name" : "gcs-secret" , "type" : "gcs" } }, "storageClassName" : "gp2" , "tenants" : { "mode" : "openshift-logging" } } } loki_resource.create(resource) cluster.restart_insights_operator() 6. Restart IO 7. View the Insights archive that path is `insights-operator/gather.json`, check whether the `clusterconfig/lokistack` filed contains the error info about `found resource in an unexpected namespace` and `found 22 resources, limit (20) reached` 8. View the relevant archives to confirm whether their contents contain the corresponding info of lokiStack resources. Expected results 1.Both of these 2 operators are installed successfully. 7. The `clusterconfig/lokistack` filed of `gather.json` archive contains the error info about `found resource in an unexpected namespace` and `found 22 resources, limit (20) reached` 8. The relevant insights archives contain the corresponding info of lokiStack resources. Actual results 1.Both of these 2 operators are installed successfully. 7. The `clusterconfig/lokistack` filed of `gather.json` archive contains the error info about `found resource in an unexpected namespace` and `found 22 resources, limit (20) reached` 8. The relevant insights archives contain the corresponding info of lokiStack resources. Remark Reference for installing operator: https://docs.redhat.com/en/documentation/openshift_container_platform/4.10/html/logging/cluster-logging-loki#logging-loki-deploy_cluster-logging-loki Reference for getting cloud platform certificates: https://gitlab.cee.redhat.com/aosqe/aosqe-tools/-/tree/master/logging/log_template/loki-operator?ref_type=heads

            OpenShift Jira Bot added a comment -

            Hi jsegural,

            Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            OpenShift Jira Bot added a comment - Hi jsegural , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

              jsegural Jose Luis Segura Lucas
              jsegural Jose Luis Segura Lucas
              baiyang zhou baiyang zhou
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: