cqu-mac:tests changliangqu$ ginkgo --focus="config/g2" -v – -options=../../resources/options.yaml -v=5
I0511 18:18:57.766862   61051 utils.go:211] Kubeconfig path
I0511 18:18:57.768268   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url  using /Users/changliangqu/.kube/config
Running Suite: Observability E2E Suite
======================================
Random Seed: 1683800328
Will run 1 of 62 specs

I0511 18:18:57.784142   61051 observability-e2e-test_suite_test.go:158] options filename=../../resources/options.yaml
file preview: ../../resources/options.yaml
I0511 18:18:57.785445   61051 observability-e2e-test_suite_test.go:198] ownerPrefix=changliangqu
I0511 18:18:57.785454   61051 observability-e2e-test_suite_test.go:205] ocpRelease=4.4.4
SSSSSSSSSSSSSSSSSSSSSSSS
------------------------------

  RHACM4K-3419: Observability: Persist advance values in MCO CR - Checking resources in advanced config [P2][Sev2][Observability][Integration] (config/g2)
  /Users/changliangqu/ACM/observability/observability_core_automation/tests/pkg/tests/observability_config_test.go:276
I0511 18:18:57.786869   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:18:57.786882   61051 utils.go:211] Kubeconfig path
I0511 18:18:57.786888   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:18:57.789982   61051 utils.go:73] Create kubeclient dynamic for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:18:57.789995   61051 utils.go:211] Kubeconfig path
I0511 18:18:57.789999   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:18:58.640130   61051 observability_config_test.go:290] The component is: rule
I0511 18:18:58.640223   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:18:58.640232   61051 utils.go:211] Kubeconfig path
I0511 18:18:58.640238   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:18:58.925431   61051 observability_config_test.go:290] The component is: alertmanager
I0511 18:18:58.925449   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:18:58.925463   61051 utils.go:211] Kubeconfig path
I0511 18:18:58.925481   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:18:59.199385   61051 observability_config_test.go:290] The component is: rbacQueryProxy
I0511 18:18:59.199542   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:18:59.199561   61051 utils.go:211] Kubeconfig path
I0511 18:18:59.199577   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:18:59.202663   61051 mco_deployments.go:29] Get get deployment with label selector <app=rbac-query-proxy> in namespace <open-cluster-management-observability>, isHub: <true>
I0511 18:18:59.481945   61051 observability_config_test.go:290] The component is: query
I0511 18:18:59.481969   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:18:59.481975   61051 utils.go:211] Kubeconfig path
I0511 18:18:59.481980   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:18:59.482894   61051 mco_deployments.go:29] Get get deployment with label selector <app.kubernetes.io/name=thanos-query> in namespace <open-cluster-management-observability>, isHub: <true>
I0511 18:18:59.755421   61051 observability_config_test.go:290] The component is: queryFrontend
I0511 18:18:59.755439   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:18:59.755443   61051 utils.go:211] Kubeconfig path
I0511 18:18:59.755448   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:18:59.756447   61051 mco_deployments.go:29] Get get deployment with label selector <app.kubernetes.io/name=thanos-query-frontend> in namespace <open-cluster-management-observability>, isHub: <true>
I0511 18:19:00.026733   61051 observability_config_test.go:290] The component is: receive
I0511 18:19:00.028760   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:19:00.028777   61051 utils.go:211] Kubeconfig path
I0511 18:19:00.028784   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:19:04.882988   61051 utils.go:73] Create kubeclient dynamic for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:19:04.883024   61051 utils.go:211] Kubeconfig path
I0511 18:19:04.883038   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:19:05.156873   61051 mco_deploy.go:192] MCO spec: {
  "advanced": {
    "alertmanager": {
      "replicas": 2,
      "resources": {
        "limits":

{           "cpu": "100m",           "memory": "400Mi"         }

      }
    },
    "compact": {
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "2Gi"         }

      },
      "serviceAccountAnnotations":

{         "test.com/role-arn": "s3_role"       }

    },
    "grafana": {
      "replicas": 3,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "1Gi"         }

      }
    },
    "observatoriumAPI": {
      "replicas": 3,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "1Gi"         }

      }
    },
    "query": {
      "replicas": 2,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "1Gi"         }

      },
      "serviceAccountAnnotations":

{         "test.com/role-arn": "s3_role"       }

    },
    "queryFrontend": {
      "replicas": 3,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "1Gi"         }

      }
    },
    "queryFrontendMemcached": {
      "connectionLimit": 2048,
      "maxItemSize": "10m",
      "memoryLimitMb": 4096,
      "replicas": 3,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "2Gi"         }

      }
    },
    "rbacQueryProxy": {
      "replicas": 3,
      "resources": {
        "limits":

{           "cpu": "50m",           "memory": "200Mi"         }

      }
    },
    "receive": {
      "replicas": 3,
      "serviceAccountAnnotations":

{         "test.com/role-arn": "s3_role"       }

    },
    "retentionConfig":

{       "blockDuration": "3h",       "deleteDelay": "50h",       "retentionInLocal": "5d",       "retentionResolution1h": "31d",       "retentionResolution5m": "15d",       "retentionResolutionRaw": "6d"     }

,
    "rule": {
      "replicas": 1,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "1Gi"         }

      },
      "serviceAccountAnnotations":

{         "test.com/role-arn": "s3_role"       }

    },
    "store": {
      "replicas": 3,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "2Gi"         }

      },
      "serviceAccountAnnotations":

{         "test.com/role-arn": "s3_role"       }

    },
    "storeMemcached": {
      "connectionLimit": 2048,
      "maxItemSize": "10m",
      "memoryLimitMb": 4096,
      "replicas": 2,
      "resources": {
        "limits":

{           "cpu": 1,           "memory": "2Gi"         }

      }
    }
  },
  "enableDownsampling": true,
  "imagePullSecret": "multiclusterhub-operator-pull-secret",
  "nodeSelector":

{     "kubernetes.io/os": "linux"   }

,
  "observabilityAddonSpec": {
    "enableMetrics": true,
    "interval": 30,
    "resources": {
      "limits":

{         "cpu": "200m",         "memory": "700Mi"       }

,
      "requests":

{         "cpu": "10m",         "memory": "100Mi"       }

    }
  },
  "storageConfig": {
    "alertmanagerStorageSize": "2Gi",
    "compactStorageSize": "2Gi",
    "metricObjectStorage":

{       "key": "thanos.yaml",       "name": "thanos-object-storage"     }

,
    "receiveStorageSize": "2Gi",
    "ruleStorageSize": "2Gi",
    "storageClass": "gp2",
    "storeStorageSize": "2Gi"
  }
}
I0511 18:19:05.156969   61051 mco_deploy.go:193] MCO status: {
  "conditions": [
   

{       "lastTransitionTime": "2023-05-11T10:12:30Z",       "message": "Installation is in progress",       "reason": "Installing",       "status": "True",       "type": "Installing"     }

,
   

{       "lastTransitionTime": "2023-05-11T10:13:35Z",       "message": "Observability components are deployed and running",       "reason": "Ready",       "status": "True",       "type": "Ready"     }

  ]
}
I0511 18:19:05.157197   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:19:05.157211   61051 utils.go:211] Kubeconfig path
I0511 18:19:05.157231   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:19:05.713474   61051 mco_deploy.go:161] Get <31> pods in <open-cluster-management-observability> namespace
I0511 18:19:05.713554   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:19:05.713586   61051 utils.go:211] Kubeconfig path
I0511 18:19:05.713597   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:19:06.253907   61051 mco_deploy.go:232] Get <3> pods in <open-cluster-management-addon-observability> namespace from managedcluster

•! Panic [8.468 seconds]

/Users/changliangqu/ACM/observability/observability_core_automation/tests/pkg/tests/observability_config_test.go:22
  RHACM4K-3419: Observability: Persist advance values in MCO CR - Checking resources in advanced config [P2][Sev2][Observability][Integration] (config/g2) [It]
  /Users/changliangqu/ACM/observability/observability_core_automation/tests/pkg/tests/observability_config_test.go:276

  Test Panicked
  interface conversion: interface {} is nil, not map[string]interface {}
  /usr/local/go/src/runtime/iface.go:262

  Full Stack Trace
  github.com/stolostron/multicluster-observability-operator/tests/pkg/tests.glob..func6.7()
      /Users/changliangqu/ACM/observability/observability_core_automation/tests/pkg/tests/observability_config_test.go:292 +0xf8c
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0008e12e8?)
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/leafnodes/runner.go:113 +0xb1
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0x0?)
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/leafnodes/runner.go:64 +0x125
  github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc00078d700?)
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/leafnodes/it_node.go:26 +0x7b
  github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc0009de1e0, 0xc0008e19e0?, {0x1018d98a0, 0xc0001788c0})
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/spec/spec.go:215 +0x28a
  github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc0009de1e0, {0x1018d98a0, 0xc0001788c0})
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/spec/spec.go:138 +0xe7
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc00017c580, 0xc0009de1e0)
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/specrunner/spec_runner.go:200 +0xe8
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc00017c580)
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/specrunner/spec_runner.go:170 +0x1a5
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc00017c580)
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/specrunner/spec_runner.go:66 +0xc5
  github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc00020e9a0, {0x134b09428, 0xc000474000}, {0x101675529, 0x17}, {0xc000266020, 0x2, 0x2}, {0x1018f1ee8, 0xc0001788c0}, ...)
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/internal/suite/suite.go:79 +0x4d2
  github.com/onsi/ginkgo.runSpecsWithCustomReporters({0x1018db300?, 0xc000474000}, {0x101675529, 0x17}, {0xc000266000, 0x2, 0x101677b1c?})
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/ginkgo_dsl.go:238 +0x189
  github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters({0x1018db300, 0xc000474000}, {0x101675529, 0x17}, {0xc000083f50, 0x1, 0x1})
      /Users/changliangqu/gopath/pkg/mod/github.com/onsi/ginkgo@v1.16.4/ginkgo_dsl.go:221 +0x1b6
  github.com/stolostron/multicluster-observability-operator/tests/pkg/tests.TestObservabilityE2E(0x0?)
      /Users/changliangqu/ACM/observability/observability_core_automation/tests/pkg/tests/observability-e2e-test_suite_test.go:129 +0x10a
  testing.tRunner(0xc000474000, 0x10174d898)
      /usr/local/go/src/testing/testing.go:1439 +0x102
  created by testing.(*T).Run
      /usr/local/go/src/testing/testing.go:1486 +0x35f
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSI0511 18:19:06.257465   61051 utils.go:58] Create kubeclient for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using kubeconfig path
I0511 18:19:06.257475   61051 utils.go:211] Kubeconfig path
I0511 18:19:06.257481   61051 utils.go:231] clientcmd.BuildConfigFromFlags for url https://api.acmqe2-o413-crun.az.dev09.red-chesterfield.com:6443 using /Users/changliangqu/.kube/config
I0511 18:19:06.873854   61051 mco_deploy.go:161] Get <31> pods in <open-cluster-management-observability> namespace

JUnit report was created: /Users/changliangqu/ACM/observability/observability_core_automation/tests/pkg/tests/results.xml

Summarizing 1 Failure:

[Panic!]  [It] RHACM4K-3419: Observability: Persist advance values in MCO CR - Checking resources in advanced config [P2][Sev2][Observability][Integration] (config/g2)
/usr/local/go/src/runtime/iface.go:262

Ran 1 of 62 Specs in 9.092 seconds
FAIL! – 0 Passed | 1 Failed | 0 Pending | 61 Skipped