Login successful. You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Login successful. You have access to 82 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". + oc config view --minify --raw=XXXX + //cat userfile ./execute_obs_interop_commands.sh: line 50: //cat: No such file or directory + whoami 1009930000 + rm -rf /alabama/.kube/config + cp userfile /alabama/.kube/config + //cat /alabama/.kube/config ./execute_obs_interop_commands.sh: line 54: //cat: No such file or directory + export KUBECONFIG=/alabama/.kube/config + KUBECONFIG=/alabama/.kube/config + go mod vendor go: downloading github.com/go-kit/kit v0.11.0 go: downloading github.com/prometheus/prometheus v1.8.2-0.20210811141203-dcb07e8eac34 go: downloading github.com/oklog/run v1.1.0 go: downloading github.com/prometheus/common v0.30.0 go: downloading github.com/go-kit/log v0.1.0 go: downloading sigs.k8s.io/controller-runtime v0.10.0 go: downloading sigs.k8s.io/kustomize/v3 v3.3.1 go: downloading open-cluster-management.io/api v0.0.0-20210916013819-2e58cdb938f9 go: downloading github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.48.1 go: downloading open-cluster-management.io/addon-framework v0.0.0-20211014025435-1f42884cdd53 go: downloading github.com/IBM/controller-filtered-cache v0.3.3 go: downloading github.com/cenkalti/backoff v2.2.1+incompatible go: downloading github.com/openshift/api v0.0.0-20210331193751-3acddb19d360 go: downloading github.com/openshift/cluster-monitoring-operator v0.1.1-0.20210611103744-7168290cd660 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/cloudflare/cfssl v1.6.0 go: downloading github.com/stolostron/multicloud-operators-foundation v0.0.0-20220112005209-c7d642306a99 go: downloading github.com/stolostron/multiclusterhub-operator v0.0.0-20220106205009-2af6f43fd562 go: downloading github.com/stolostron/observatorium-operator v0.0.0-20220307015247-f9eb849e218e go: downloading github.com/prometheus-operator/prometheus-operator/pkg/client v0.47.1 go: downloading k8s.io/kubectl v0.21.2 go: downloading github.com/prometheus-community/prom-label-proxy v0.3.0 go: downloading sigs.k8s.io/kustomize/api v0.8.8 go: downloading github.com/openshift/kubernetes-kube-storage-version-migrator v0.0.3-0.20210302135122-481bd04dbc78 go: downloading github.com/hashicorp/go-version v1.3.0 go: downloading github.com/onsi/ginkgo v1.16.4 go: downloading github.com/onsi/gomega v1.16.0 go: downloading github.com/prometheus/alertmanager v0.22.2 go: downloading github.com/google/uuid v1.2.0 go: downloading k8s.io/klog/v2 v2.10.0 go: downloading github.com/inconshreveable/mousetrap v1.0.0 go: downloading github.com/google/gofuzz v1.2.0 go: downloading golang.org/x/sys v0.0.0-20210817190340-bfb29a6856f2 go: downloading golang.org/x/net v0.0.0-20211209124913-491a49abca63 go: downloading github.com/go-logfmt/logfmt v0.5.0 go: downloading github.com/gobuffalo/flect v0.2.2 go: downloading gomodules.xyz/jsonpatch/v2 v2.2.0 go: downloading k8s.io/utils v0.0.0-20210802155522-efc7438f0176 go: downloading github.com/google/go-cmp v0.5.6 go: downloading github.com/go-logr/zapr v0.4.0 go: downloading go.uber.org/zap v1.19.0 go: downloading github.com/brancz/locutus v0.0.0-20210511124350-7a84f4d1bcb3 go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: downloading golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914 go: downloading github.com/zmap/zlint/v3 v3.0.0 go: downloading github.com/google/certificate-transparency-go v1.0.21 go: downloading github.com/zmap/zcrypto v0.0.0-20201128221613-3719af1573cf go: downloading google.golang.org/protobuf v1.27.1 go: downloading k8s.io/apiserver v0.22.1 go: downloading github.com/efficientgo/tools/core v0.0.0-20210201224146-3d78f4d30648 go: downloading github.com/go-openapi/runtime v0.19.28 go: downloading github.com/go-openapi/strfmt v0.20.1 go: downloading sigs.k8s.io/kustomize/kyaml v0.10.17 go: downloading k8s.io/kube-openapi v0.0.0-20210305001622-591a79e4bda7 go: downloading github.com/Azure/go-autorest/autorest v0.11.19 go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.14 go: downloading github.com/fsnotify/fsnotify v1.5.1 go: downloading github.com/jpillora/backoff v1.0.0 go: downloading go.uber.org/atomic v1.9.0 go: downloading go.uber.org/multierr v1.6.0 go: downloading golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e go: downloading github.com/jmoiron/sqlx v1.3.1 go: downloading github.com/Azure/go-autorest v14.2.0+incompatible go: downloading github.com/weppos/publicsuffix-go v0.13.0 go: downloading cloud.google.com/go v0.87.0 go: downloading google.golang.org/appengine v1.6.7 go: downloading github.com/go-openapi/spec v0.19.5 go: downloading github.com/asaskevich/govalidator v0.0.0-20200907205600-7a23bdc65eef go: downloading github.com/go-openapi/errors v0.20.0 go: downloading github.com/mitchellh/mapstructure v1.4.1 go: downloading github.com/oklog/ulid v1.3.1 go: downloading go.mongodb.org/mongo-driver v1.5.1 go: downloading github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15 go: downloading github.com/edsrzf/mmap-go v1.0.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/uber/jaeger-client-go v2.29.1+incompatible go: downloading github.com/nxadm/tail v1.4.8 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/go-openapi/swag v0.19.15 go: downloading github.com/go-openapi/validate v0.20.2 go: downloading golang.org/x/text v0.3.5 go: downloading github.com/Azure/go-autorest/logger v0.2.1 go: downloading github.com/Azure/go-autorest/tracing v0.6.0 go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0 go: downloading github.com/form3tech-oss/jwt-go v3.2.5+incompatible go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c go: downloading go.uber.org/goleak v1.1.10 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/emicklei/go-restful v2.14.2+incompatible go: downloading github.com/xlab/treeprint v1.1.0 go: downloading github.com/dennwc/varint v1.0.0 go: downloading github.com/go-openapi/analysis v0.19.5 go: downloading github.com/go-openapi/loads v0.19.5 go: downloading golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 go: downloading golang.org/x/tools v0.1.5 go: downloading github.com/go-stack/stack v1.8.0 + ginkgo build ./tests/pkg/tests/ Compiling tests... compiled tests.test + cd tests + cp resources/options.yaml.template resources/options.yaml + /usr/local/bin/yq e -i '.options.hub.name="ci-op-id52jm5k-51b43"' resources/options.yaml + /usr/local/bin/yq e -i '.options.hub.baseDomain="ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com"' resources/options.yaml + /usr/local/bin/yq e -i '.options.clusters.name="acm-interops-opp-ci-gcp-1691390716081"' resources/options.yaml + /usr/local/bin/yq e -i '.options.clusters.baseDomain="acm-interops-opp-ci-gcp-1691390716081.gcp.dev09.red-chesterfield.com"' resources/options.yaml + /usr/local/bin/yq e -i '.options.clusters.kubeconfig="/alabama/.kube/managed_kubeconfig"' resources/options.yaml + cat resources/options.yaml options: hub: name: ci-op-id52jm5k-51b43 baseDomain: ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com clusters: name: acm-interops-opp-ci-gcp-1691390716081 baseDomain: acm-interops-opp-ci-gcp-1691390716081.gcp.dev09.red-chesterfield.com kubeconfig: /alabama/.kube/managed_kubeconfig + ginkgo -v pkg/tests/ -- -options=../../resources/options.yaml -v=5 I0807 07:30:12.179425 7951 utils.go:211] Kubeconfig path /alabama/.kube/config Running Suite: Observability E2E Suite ====================================== Random Seed: 1691393411 Will run 62 of 62 specs I0807 07:30:12.182541 7951 observability-e2e-test_suite_test.go:158] options filename=../../resources/options.yaml file preview: ../../resources/options.yaml E0807 07:30:12.182667 7951 observability-e2e-test_suite_test.go:170] --options error: yaml: unmarshal errors: line 6: cannot unmarshal !!map into []utils.Cluster I0807 07:30:12.182698 7951 observability-e2e-test_suite_test.go:198] ownerPrefix=ginkgo I0807 07:30:12.182702 7951 observability-e2e-test_suite_test.go:205] ocpRelease=4.4.4 I0807 07:30:12.182707 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.182713 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.183519 7951 utils.go:73] Create kubeclient dynamic for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.183531 7951 utils.go:211] Kubeconfig path /alabama/.kube/config STEP: Deploy CM cluster-monitoring-config I0807 07:30:12.190408 7951 utils.go:256] obj:map[] I0807 07:30:12.190481 7951 utils.go:269] kind: ConfigMap I0807 07:30:12.190491 7951 utils.go:277] apiVersion: v1 I0807 07:30:12.190493 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.190496 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.191097 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.191109 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.191517 7951 utils.go:374] Install ConfigMap: apiVersion: v1 data: config.yaml: |- alertmanagerMain: enableUserAlertmanagerConfig: XXXX kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring W0807 07:30:12.262434 7951 utils.go:389] ConfigMap openshift-monitoring/cluster-monitoring-config already exists, updating! STEP: Checking MCO operator is started up and running STEP: Checking Required CRDs are created I0807 07:30:12.336544 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.336557 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.337114 7951 utils.go:636] Check if multiclusterobservabilities.observability.open-cluster-management.io exists I0807 07:30:12.368700 7951 utils.go:636] Check if observatoria.core.observatorium.io exists I0807 07:30:12.402912 7951 utils.go:636] Check if observabilityaddons.observability.open-cluster-management.io exists I0807 07:30:12.418049 7951 mco_deploy.go:798] Create open-cluster-management-observability namespaces I0807 07:30:12.418064 7951 utils.go:256] obj:map[] I0807 07:30:12.418131 7951 utils.go:269] kind: Namespace I0807 07:30:12.418140 7951 utils.go:277] apiVersion: v1 I0807 07:30:12.418143 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.418146 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.418971 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.418982 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.419477 7951 utils.go:304] Install Namespace: apiVersion: v1 kind: Namespace metadata: name: open-cluster-management-observability I0807 07:30:12.454094 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.454106 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.454770 7951 utils.go:73] Create kubeclient dynamic for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.454781 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.542758 7951 mco_deploy.go:778] Delete existing pullSecret - secrets "multiclusterhub-operator-pull-secret" not found I0807 07:30:12.542780 7951 mco_deploy.go:785] Create MCO pull secret I0807 07:30:12.561410 7951 mco_deploy.go:849] Create MCO object storage secret I0807 07:30:12.561422 7951 utils.go:256] obj:map[] I0807 07:30:12.561499 7951 utils.go:269] kind: Secret I0807 07:30:12.561508 7951 utils.go:277] apiVersion: v1 I0807 07:30:12.561511 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.561513 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.562551 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:12.562566 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:12.563327 7951 utils.go:357] Install Secret: apiVersion: v1 kind: Secret metadata: name: thanos-object-storage namespace: open-cluster-management-observability stringData: thanos.yaml: | type: s3 config: bucket: obs-v1 endpoint: s3.us-east-1.amazonaws.com insecure: false access_key: XXXXXXXXXXXXXXXXXXXX secret_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX type: Opaque I0807 07:30:13.251846 7951 utils.go:256] obj:map[] I0807 07:30:13.251934 7951 utils.go:269] kind: LimitRange I0807 07:30:13.251944 7951 utils.go:277] apiVersion: v1 I0807 07:30:13.251948 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.251957 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.252586 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.252597 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.253037 7951 utils.go:452] Install LimitRange: apiVersion: v1 kind: LimitRange metadata: name: limit-range namespace: open-cluster-management-observability spec: limits: - defaultRequest: cpu: 10m memory: 50Mi max: cpu: "5.25" memory: 12Gi type: Container I0807 07:30:13.285642 7951 utils.go:256] obj:map[] I0807 07:30:13.285786 7951 utils.go:269] kind: ResourceQuota I0807 07:30:13.285813 7951 utils.go:277] apiVersion: v1 I0807 07:30:13.285818 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.285828 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.287570 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.287582 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.288300 7951 utils.go:471] Install ResourceQuota: apiVersion: v1 kind: ResourceQuota metadata: name: resource-quota namespace: open-cluster-management-observability spec: hard: cpu: "5.25" memory: 15Gi STEP: Creating the MCO testing RBAC resources I0807 07:30:13.320638 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.320647 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.338306 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.338319 7951 utils.go:211] Kubeconfig path /alabama/.kube/config STEP: Creating MCO instance of v1beta1 I0807 07:30:13.364946 7951 utils.go:256] obj:map[] I0807 07:30:13.365031 7951 utils.go:269] kind: MultiClusterObservability I0807 07:30:13.365040 7951 utils.go:277] apiVersion: observability.open-cluster-management.io/v1beta1 I0807 07:30:13.365043 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.365045 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.365615 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.365626 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.366047 7951 utils.go:514] Install MultiClusterObservability: apiVersion: observability.open-cluster-management.io/v1beta1 kind: MultiClusterObservability metadata: annotations: null name: observability spec: nodeSelector: kubernetes.io/os: linux observabilityAddonSpec: {} retentionResolution1h: 30d retentionResolution5m: 14d retentionResolutionRaw: 5d storageConfigObject: metricObjectStorage: key: thanos.yaml name: thanos-object-storage statefulSetSize: 2Gi statefulSetStorageClass: gp2 I0807 07:30:13.366064 7951 utils.go:73] Create kubeclient dynamic for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.366068 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:30:13.400382 7951 utils.go:73] Create kubeclient dynamic for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:30:13.400394 7951 utils.go:211] Kubeconfig path /alabama/.kube/config W0807 07:30:13.441354 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:13.476122 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability STEP: Waiting for MCO ready status W0807 07:30:13.492242 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:18.509305 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:23.527992 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:28.545353 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:33.563433 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:38.581004 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:43.598653 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:48.616302 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:53.650858 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:30:58.674366 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:03.693252 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:08.711270 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:13.728743 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:18.747080 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:23.765590 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:28.783799 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:33.802344 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:38.819963 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:43.837554 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:48.858029 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability W0807 07:31:53.876676 7951 warnings.go:70] observability.open-cluster-management.io/v1beta1 MultiClusterObservability is deprecated in v2.3+, unavailable in v2.6+; use observability.open-cluster-management.io/v1beta2 MultiClusterObservability STEP: Check clustermanagementaddon CR is created STEP: Check the api conversion is working as expected I0807 07:31:53.892246 7951 utils.go:73] Create kubeclient dynamic for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:31:53.892257 7951 utils.go:211] Kubeconfig path /alabama/.kube/config STEP: Apply MCO instance of v1beta2 I0807 07:31:53.914547 7951 utils.go:256] obj:map[] I0807 07:31:53.914914 7951 utils.go:269] kind: MultiClusterObservability I0807 07:31:53.914924 7951 utils.go:277] apiVersion: observability.open-cluster-management.io/v1beta2 I0807 07:31:53.914929 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:31:53.914933 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:31:53.915546 7951 utils.go:88] Create kubeclient apiextension for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:31:53.915557 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:31:53.916083 7951 utils.go:514] Install MultiClusterObservability: apiVersion: observability.open-cluster-management.io/v1beta2 kind: MultiClusterObservability metadata: annotations: null name: observability spec: advanced: alertmanager: replicas: 2 resources: limits: cpu: 100m memory: 400Mi compact: resources: limits: cpu: 1 memory: 2Gi serviceAccountAnnotations: test.com/role-arn: s3_role grafana: replicas: 3 resources: limits: cpu: 1 memory: 1Gi observatoriumAPI: replicas: 3 resources: limits: cpu: 1 memory: 1Gi query: replicas: 3 resources: limits: cpu: 1 memory: 1Gi serviceAccountAnnotations: test.com/role-arn: s3_role queryFrontend: replicas: 3 resources: limits: cpu: 1 memory: 1Gi queryFrontendMemcached: connectionLimit: 2048 maxItemSize: 2m memoryLimitMb: 2048 replicas: 3 resources: limits: cpu: 1 memory: 2Gi rbacQueryProxy: replicas: 3 resources: limits: cpu: 50m memory: 200Mi receive: replicas: 2 resources: limits: cpu: 1 memory: 4Gi serviceAccountAnnotations: test.com/role-arn: s3_role retentionConfig: blockDuration: 3h cleanupInterval: 6m deleteDelay: 50h retentionInLocal: 5d retentionResolution1h: 31d retentionResolution5m: 15d retentionResolutionRaw: 6d rule: replicas: 1 resources: limits: cpu: 1 memory: 1Gi serviceAccountAnnotations: test.com/role-arn: s3_role store: replicas: 3 resources: limits: cpu: 1 memory: 2Gi serviceAccountAnnotations: test.com/role-arn: s3_role storeMemcached: connectionLimit: 2048 maxItemSize: 2m memoryLimitMb: 2048 replicas: 2 resources: limits: cpu: 1 memory: 2Gi nodeSelector: kubernetes.io/os: linux observabilityAddonSpec: enableMetrics: XXXX interval: 30 resources: limits: cpu: 200m memory: 700Mi requests: cpu: 10m memory: 100Mi storageConfig: alertmanagerStorageSize: 2Gi compactStorageSize: 2Gi metricObjectStorage: key: thanos.yaml name: thanos-object-storage receiveStorageSize: 2Gi ruleStorageSize: 2Gi storageClass: gp2 storeStorageSize: 2Gi I0807 07:31:53.916094 7951 utils.go:73] Create kubeclient dynamic for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:31:53.916099 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:31:53.951263 7951 utils.go:73] Create kubeclient dynamic for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:31:53.951276 7951 utils.go:211] Kubeconfig path /alabama/.kube/config W0807 07:31:53.967283 7951 utils.go:559] MultiClusterObservability observability already exists, updating! W0807 07:31:54.008301 7951 warnings.go:70] unknown field "spec.advanced.retentionConfig.cleanupInterval" STEP: Waiting for MCO ready status I0807 07:32:54.010526 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:32:54.010542 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:32:54.195043 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:32:54.195060 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:32:54.234035 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:33:04.235725 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:04.235745 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:04.399867 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:04.399883 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:04.438088 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:33:14.439734 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:14.439756 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:14.598043 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:14.598058 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:14.635109 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:33:24.636755 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:24.636777 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:24.799107 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:24.799125 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:24.839643 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:33:34.840731 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:34.840756 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:34.985064 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:34.985079 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:35.023951 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:33:45.026657 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:45.026696 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:45.168340 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:45.168356 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:45.205851 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:33:55.208731 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:55.208754 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:55.392774 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:33:55.392790 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:33:55.437005 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:34:05.439754 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:05.439774 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:05.592117 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:05.592130 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:05.633245 7951 mco_deploy.go:161] Get <32> pods in namespace I0807 07:34:15.636756 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:15.636777 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:15.806201 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:15.806248 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:15.847266 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:34:25.849744 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:25.849765 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:26.012915 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:26.012933 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:26.050764 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:34:36.053768 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:36.053795 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:36.215932 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:36.215956 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:36.256854 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:34:46.260816 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:46.260844 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:46.419363 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:46.419380 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:46.458326 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:34:56.461735 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:56.461754 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:56.651731 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:34:56.651752 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:34:56.688787 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:35:06.692738 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:06.692760 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:06.852934 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:06.852951 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:06.890798 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:35:16.895520 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:16.895541 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:17.059663 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:17.059698 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:17.101181 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:35:27.104730 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:27.104751 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:27.262490 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:27.262504 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:27.299609 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:35:37.299675 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:37.299733 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:37.461609 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:37.461626 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:37.499368 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:35:47.500178 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:47.500198 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:47.658413 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:47.658430 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:47.694869 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:35:57.695314 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:57.695335 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:57.854303 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:35:57.854317 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:35:57.892565 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:36:07.893673 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:07.893707 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:08.050384 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:08.050401 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:08.087616 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:36:18.087773 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:18.087796 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:18.248177 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:18.248193 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:18.286032 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:36:28.286735 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:28.286757 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:28.442573 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:28.442588 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:28.480808 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:36:38.482098 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:38.482120 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:38.641539 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:38.641552 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:38.679105 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:36:48.679742 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:48.679773 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:48.845124 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:48.845138 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:48.882591 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:36:58.882731 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:58.882751 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:59.077109 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:36:59.077124 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:36:59.114777 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:37:09.115551 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:09.115569 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:09.274292 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:09.274309 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:09.311394 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:37:19.312962 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:19.312993 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:19.471531 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:19.471547 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:19.511467 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:37:29.512733 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:29.512759 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:29.672303 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:29.672319 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:29.711048 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:37:39.713428 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:39.713463 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:39.871502 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:39.871518 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:39.909926 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:37:49.911750 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:49.911772 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:50.073151 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:37:50.073172 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:37:50.113812 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:38:00.116758 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:00.116784 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:00.279121 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:00.279138 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:00.317890 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:38:10.320753 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:10.320777 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:10.495727 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:10.495743 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:10.536239 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:38:20.536886 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:20.536905 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:20.696369 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:20.696386 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:20.733212 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:38:30.735741 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:30.735766 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:30.895261 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:30.895276 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:30.932328 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:38:40.935729 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:40.935750 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:41.095249 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:41.095266 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:41.132735 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:38:51.135761 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:51.135794 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:51.296236 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:38:51.296267 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:38:51.334261 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:39:01.338304 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:01.338336 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:01.519518 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:01.519545 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:01.556417 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:39:11.559759 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:11.559790 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:11.718976 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:11.718990 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:11.756252 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:39:21.760924 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:21.760949 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:21.920938 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:21.920958 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:21.959587 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:39:31.962771 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:31.962803 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:32.121326 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:32.121343 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:32.158941 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:39:42.159864 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:42.159884 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:42.319612 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:42.319625 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:42.357116 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:39:52.358044 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:52.358063 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:52.520113 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:39:52.520129 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:39:52.558471 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:40:02.558745 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:02.558771 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:02.719158 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:02.719176 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:02.756291 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:40:12.756446 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:12.756476 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:12.915913 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:12.915934 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:12.954195 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:40:22.954602 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:22.954634 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:23.114656 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:23.114675 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:23.152650 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:40:33.152737 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:33.152768 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:33.315845 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:33.315860 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:33.367830 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:40:43.368745 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:43.368770 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:43.543348 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:43.543363 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:43.585985 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:40:53.586802 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:53.586822 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:53.746121 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:40:53.746142 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:40:53.783925 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:41:03.784753 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:03.784777 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:03.969019 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:03.969037 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:04.006162 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:41:14.006759 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:14.006786 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:14.164740 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:14.164756 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:14.201902 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:41:24.203832 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:24.203852 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:24.369211 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:24.369226 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:24.407107 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:41:34.408742 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:34.408768 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:34.570074 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:34.570093 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:34.610075 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:41:44.611734 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:44.611760 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:44.772928 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:44.772942 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:44.813572 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:41:54.814751 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:54.814776 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:54.975136 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:41:54.975153 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:41:55.014130 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:42:05.016620 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:05.016639 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:05.174442 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:05.174456 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:05.211827 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:42:15.214733 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:15.214755 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:15.378454 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:15.378473 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:15.416154 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:42:25.418770 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:25.418797 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:25.587851 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:25.587868 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:25.627115 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:42:35.627738 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:35.627760 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:35.783343 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:35.783361 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:35.820648 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:42:45.822784 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:45.822831 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:45.985592 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:45.985603 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:46.024005 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:42:56.027654 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:56.027674 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:56.188357 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:42:56.188374 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:42:56.225933 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:43:06.228754 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:06.228785 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:06.414879 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:06.414894 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:06.452836 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:43:16.456745 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:16.456768 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:16.613526 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:16.613539 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:16.651456 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:43:26.655303 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:26.655338 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:26.816372 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:26.816393 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:26.855349 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:43:36.858727 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:36.858746 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:37.021439 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:37.021459 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:37.062162 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:43:47.065730 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:47.065750 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:47.234246 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:47.234265 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:47.271479 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:43:57.271963 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:57.271984 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:57.456581 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:43:57.456596 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:43:57.495927 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:44:07.496165 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:07.496184 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:07.659509 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:07.659526 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:07.697304 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:44:17.697390 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:17.697425 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:17.870058 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:17.870077 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:17.908928 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:44:27.909532 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:27.909558 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:28.069160 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:28.069176 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:28.109792 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:44:38.110574 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:38.110606 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:38.267646 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:38.267661 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:38.304410 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:44:48.304742 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:48.304765 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:48.477727 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:48.477744 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:48.518856 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:44:58.519748 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:58.519780 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:58.680833 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:44:58.680853 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:44:58.721341 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:45:08.721729 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:08.721749 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:08.910411 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:08.910429 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:08.947952 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:45:18.948770 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:18.948800 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:19.106127 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:19.106143 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:19.144287 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:45:29.145753 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:29.145780 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:29.305846 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:29.305863 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:29.343747 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:45:39.345747 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:39.345776 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:39.506963 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:39.506984 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:39.543557 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:45:49.544733 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:49.544755 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:49.702716 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:49.702732 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:49.742031 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:45:59.743748 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:59.743768 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:59.902356 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:45:59.902373 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:45:59.940301 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:46:09.941744 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:09.941763 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:10.103737 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:10.103752 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:10.140072 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:46:20.142736 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:20.142760 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:20.307655 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:20.307671 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:20.348185 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:46:30.351044 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:30.351065 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:30.508425 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:30.508444 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:30.546287 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:46:40.548746 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:40.548771 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:40.710018 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:40.710032 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:40.750483 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:46:50.753370 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:50.753391 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:50.913940 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:46:50.913955 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:46:50.951364 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:47:00.953729 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:00.953747 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:01.110490 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:01.110508 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:01.147437 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:47:11.150786 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:11.150824 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:11.336414 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:11.336431 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:11.376492 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:47:21.379754 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:21.379788 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:21.537467 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:21.537489 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:21.575603 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:47:31.578751 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:31.578777 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:31.742409 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:31.742426 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:31.780532 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:47:41.783754 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:41.783784 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:41.955110 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:41.955138 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:41.999096 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:47:52.002745 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:52.002769 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:52.163662 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:47:52.163700 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:47:52.201695 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:48:02.202820 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:02.202853 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:02.363549 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:02.363586 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:02.402493 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:48:12.403125 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:12.403156 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:12.564370 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:12.564389 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:12.600646 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:48:22.601644 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:22.601667 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:22.763399 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:22.763414 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:22.803275 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:48:32.803432 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:32.803456 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:32.960167 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:32.960197 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:33.001704 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:48:43.002721 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:43.002743 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:43.164512 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:43.164530 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:43.201101 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:48:53.201726 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:53.201748 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:53.389222 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:48:53.389238 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:48:53.429663 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:49:03.430077 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:03.430098 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:03.592515 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:03.592530 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:03.630841 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:49:13.632374 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:13.632408 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:13.819537 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:13.819565 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:13.858882 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:49:23.859757 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:23.859781 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:24.017243 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:24.017259 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:24.054789 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:49:34.055733 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:34.055756 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:34.214376 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:34.214396 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:34.252010 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:49:44.253913 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:44.253940 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:44.415617 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:44.415634 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:44.454477 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:49:54.455739 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:54.455765 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:54.621766 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:49:54.621783 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:49:54.659947 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:50:04.662296 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:04.662317 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:04.822249 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:04.822264 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:04.860129 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:50:14.860736 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:14.860758 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:15.018762 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:15.018780 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:15.056739 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:50:25.058743 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:25.058766 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:25.216673 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:25.216709 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:25.256563 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:50:35.256951 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:35.256974 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:35.416759 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:35.416780 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:35.454070 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:50:45.456741 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:45.456763 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:45.619473 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:45.619488 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:45.656255 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:50:55.658732 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:55.658757 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:55.816167 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:50:55.816197 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:50:55.856372 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:51:05.858729 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:05.858749 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:06.033488 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:06.033514 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:06.073827 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:51:16.077723 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:16.077746 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:16.260217 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:16.260233 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:16.298500 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:51:26.301730 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:26.301752 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:26.467037 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:26.467055 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:26.509340 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:51:36.513146 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:36.513169 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:36.673017 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:36.673038 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:36.712903 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:51:46.715740 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:46.715764 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:46.872771 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:46.872785 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:46.908654 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:51:56.911752 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:56.911775 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:57.072356 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:51:57.072379 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:51:57.113026 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:52:07.117770 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:07.117809 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:07.288944 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:07.288966 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:07.333958 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:52:17.335018 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:17.335039 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:17.494874 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:17.494892 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:17.534169 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:52:27.535254 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:27.535278 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:27.693010 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:27.693026 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:27.731596 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:52:37.732430 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:37.732452 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:37.889388 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:37.889408 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:37.930455 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:52:47.930574 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:47.930593 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:48.088329 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:48.088360 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:48.127840 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:52:58.128736 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:58.128760 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:58.289825 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:52:58.289844 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:52:58.326761 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:53:08.327739 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:08.327760 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:08.486185 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:08.486207 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:08.523373 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:53:18.523537 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:18.523572 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:18.715924 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:18.715949 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:18.753853 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:53:28.754741 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:28.754765 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:28.913831 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:28.913850 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:28.951746 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:53:38.952731 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:38.952753 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:39.112401 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:39.112415 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:39.150517 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:53:49.151842 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:49.151876 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:49.370379 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:49.370396 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:49.416508 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:53:59.417736 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:59.417761 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:59.578190 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:53:59.578205 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:53:59.616986 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:54:09.617731 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:09.617752 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:09.787098 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:09.787126 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:09.823665 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:54:19.824725 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:19.824743 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:19.985498 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:19.985515 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:20.024802 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:54:30.026752 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:30.026782 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:30.191092 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:30.191111 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:30.231607 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:54:40.233747 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:40.233771 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:40.395432 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:40.395453 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:40.438295 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:54:50.440758 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:50.440785 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:50.606919 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:54:50.606936 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:54:50.644757 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:55:00.647742 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:00.647763 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:00.819396 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:00.819415 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:00.856825 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:55:10.860263 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:10.860305 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:11.032256 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:11.032270 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:11.071744 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:55:21.074756 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:21.074779 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:21.270475 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:21.270490 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:21.306598 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:55:31.306743 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:31.306764 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:31.471349 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:31.471371 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:31.508420 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:55:41.512159 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:41.512192 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:41.671029 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:41.671046 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:41.708972 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:55:51.712740 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:51.712767 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:51.873272 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:55:51.873297 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:55:51.910718 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:56:01.914735 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:01.914758 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:02.072917 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:02.072936 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:02.112659 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:56:12.116760 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:12.116785 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:12.275521 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:12.275537 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:12.311855 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:56:22.311972 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:22.311993 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:22.470127 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:22.470143 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:22.507969 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:56:32.508855 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:32.508881 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:32.673810 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:32.673830 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:32.714253 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:56:42.714353 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:42.714386 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:42.874493 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:42.874507 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:42.913048 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:56:52.913555 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:52.913575 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:53.083419 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:56:53.083442 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:56:53.124664 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:57:03.124762 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:03.124790 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:03.292736 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:03.292753 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:03.333663 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:57:13.333829 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:13.333855 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:13.494768 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:13.494785 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:13.533735 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:57:23.534739 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:23.534766 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:23.721087 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:23.721110 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:23.762045 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:57:33.763437 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:33.763459 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:33.929750 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:33.929768 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:33.967163 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:57:43.967741 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:43.967766 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:44.131348 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:44.131364 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:44.170423 7951 mco_deploy.go:161] Get <31> pods in namespace I0807 07:57:54.010917 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:54.010933 7951 utils.go:211] Kubeconfig path /alabama/.kube/config [DEBUG] MCO is installed failed, checking MCO operator logs: 1.691393482508217e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913934825130482e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.691393482515966e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913934826188958e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913934826657333e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913934826727996e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913934827050228e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913934827198672e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913934827198858e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913934827198918e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913934827199516e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913934827199688e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913934827199733e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913934827200458e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.691393482720066e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913934827201192e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913934827201273e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913934827201297e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.691393482720133e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913934827201388e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913934827201407e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913934827201426e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913934827201462e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913934827211266e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913934827211554e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913934835126529e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934855136473e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934870508885e+09 INFO controller_placementrule UpdateFunc {"managedCluster": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913934870509784e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913934870510588e+09 INFO util observability-controller clustermanagementaddon is present 1.6913934870510669e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913934870511441e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913934870511532e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913934870511575e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913934870511613e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.691393487051165e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913934870511694e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913934870511734e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913934870516748e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913934870517862e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "acm-interops-opp-ci-gcp-1691390716081", "request.namespace": ""} 1.6913934870518005e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913934870518148e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913934870518274e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913934870518408e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393487051947e+09 INFO controller_placementrule Updating manifestwork {"acm-interops-opp-ci-gcp-1691390716081": "acm-interops-opp-ci-gcp-1691390716081", "name": "acm-interops-opp-ci-gcp-1691390716081-observability"} 1.6913934875142348e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934895154076e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934915185652e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934935197012e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934955201883e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934975205433e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913934995213714e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.691393501522198e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913935035235064e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913935055241275e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.691393507525474e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913935095262296e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is triggered. 1.6913935095268304e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935095391462e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935095394628e+09 DEBUG controller_multiclustermonitoring send singal to stop status check ready goroutine because MCO status is ready 1.6913935095394797e+09 DEBUG controller_multiclustermonitoring check status ready goroutine is stopped. 1.6913935095395815e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935095395973e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935095396183e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935095396242e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935095399225e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935095399323e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.691393509539936e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935095399384e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.691393509539942e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935095399446e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935095399468e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935095403636e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935095403745e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935095403779e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935095403836e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935095403874e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935095405262e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935095405343e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935095516913e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935095544643e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935095575793e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935095641139e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935095785682e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935095818756e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.69139350973509e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935097380955e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935097411494e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.691393509778969e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935098058212e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935098204794e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.691393509831653e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935098499167e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935098499372e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935098499434e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935098500028e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935098500211e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.691393509850026e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935098500996e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935098501186e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935098501658e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.691393509850178e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935098501823e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935098501873e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935098501978e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935098502016e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935098502057e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.691393509850212e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935098510854e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935098511188e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.691393512968267e+09 INFO controller_placementrule UpdateFunc {"managedCluster": "local-cluster"} 1.691393512970594e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "local-cluster"} 1.691393512970783e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935129708164e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935129709013e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.691393512970923e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935129709392e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935129709632e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935129709787e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.691393512970994e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935129710088e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.691393512971572e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935129717216e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "local-cluster", "request.name": "local-cluster", "request.namespace": ""} 1.691393512971765e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "local-cluster"} 1.691393512971802e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935129718354e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935129718616e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "local-cluster"} 1.6913935129720163e+09 INFO controller_placementrule Updating manifestwork {"local-cluster": "local-cluster", "name": "local-cluster-observability"} 1.691393513988064e+09 DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/validate-observability-open-cluster-management-io-v1beta2-multiclusterobservability", "UID": "d7dd1c1d-9f79-4c4a-9d47-ba82a41e3dfa", "kind": "observability.open-cluster-management.io/v1beta2, Kind=MultiClusterObservability", "resource": {"group":"observability.open-cluster-management.io","version":"v1beta2","resource":"multiclusterobservabilities"}} 1.6913935139885092e+09 INFO multiclusterobservability-resource validate update {"name": "observability"} 1.6913935139962094e+09 DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/validate-observability-open-cluster-management-io-v1beta2-multiclusterobservability", "code": 200, "reason": "", "UID": "d7dd1c1d-9f79-4c4a-9d47-ba82a41e3dfa", "allowed": XXXX} 1.6913935140038395e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935140038931e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "mco-updated-request"} 1.691393514003969e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935140040007e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935140040643e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935140040932e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935140041168e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935140041351e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935140041556e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935140041754e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935140042e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935140041876e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935140043035e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.691393514004318e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.691393514004322e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935140045817e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935140047193e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "local-cluster", "request.name": "mco-updated-request", "request.namespace": ""} 1.691393514004751e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "local-cluster"} 1.6913935140047662e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935140047789e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935140047991e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "local-cluster"} 1.6913935140047235e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.691393514004863e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935140048819e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935140049045e+09 INFO controller_placementrule Updating manifestwork {"local-cluster": "local-cluster", "name": "local-cluster-observability"} 1.6913935140049067e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.691393514005049e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935140050545e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935140056405e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935140056531e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393514005658e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935140056663e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935140056696e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935140058289e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935140058372e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935140161805e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935140197947e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.691393514023315e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935140282116e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "mco-updated-request", "request.namespace": ""} 1.6913935140282652e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140282943e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514028318e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140283413e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140284429e+09 INFO controller_placementrule Updating manifestwork {"acm-interops-opp-ci-gcp-1691390716081": "acm-interops-opp-ci-gcp-1691390716081", "name": "acm-interops-opp-ci-gcp-1691390716081-observability"} 1.6913935140334854e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935140502446e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393514052793e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "local-cluster", "Request.Name": "local-cluster-observability"} 1.6913935140528667e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935140528731e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935140529206e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935140530105e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935140530481e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935140530741e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935140530953e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935140531127e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935140531294e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935140532804e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.691393514053681e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.691393514053814e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "local-cluster", "request.name": "local-cluster-observability", "request.namespace": "local-cluster"} 1.6913935140538418e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "local-cluster"} 1.6913935140538657e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935140538867e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935140539112e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "local-cluster"} 1.6913935140556674e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "local-cluster"} 1.6913935140576735e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "acm-interops-opp-ci-gcp-1691390716081", "Request.Name": "acm-interops-opp-ci-gcp-1691390716081-observability"} 1.6913935140577688e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935140577786e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.691393514057824e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935140578332e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935140578365e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935140578399e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.691393514057843e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935140578463e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935140578496e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.691393514058186e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935140582778e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "acm-interops-opp-ci-gcp-1691390716081-observability", "request.namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514058291e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140583005e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140583081e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140583174e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514059854e+09 INFO util Find updated deployment {"deployment": "endpoint-observability-operator"} 1.6913935140598664e+09 INFO controller_placementrule Updating manifestwork {"acm-interops-opp-ci-gcp-1691390716081": "acm-interops-opp-ci-gcp-1691390716081", "name": "acm-interops-opp-ci-gcp-1691390716081-observability"} 1.691393514061984e+09 INFO deploying Update {"Kind:": "apps/v1, Kind=Deployment", "Name:": "observability-grafana"} 1.6913935140907948e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "acm-interops-opp-ci-gcp-1691390716081", "Request.Name": "acm-interops-opp-ci-gcp-1691390716081-observability"} 1.691393514090872e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935140908785e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935140909293e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935140909405e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935140909448e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935140909488e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935140909536e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935140909576e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935140909617e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935140914106e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.691393514091532e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "acm-interops-opp-ci-gcp-1691390716081-observability", "request.namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140915496e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514091564e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140915763e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514091592e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935140931091e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935141744573e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "local-cluster", "Request.Name": "observability-addon"} 1.691393514174556e+09 INFO util observability-controller clustermanagementaddon is present 1.691393514174564e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.691393514174637e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935141746476e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935141746516e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935141746554e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.691393514174659e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935141746626e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935141746664e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935141751354e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935141752448e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "local-cluster", "request.name": "observability-addon", "request.namespace": "local-cluster"} 1.6913935141752598e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "local-cluster"} 1.691393514175272e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.691393514175283e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935141752956e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "local-cluster"} 1.6913935141771615e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "local-cluster"} 1.6913935142027857e+09 INFO controller_placementrule Updated status for managedclusteraddon {"namespace": "local-cluster"} 1.6913935142302473e+09 INFO controller_placementrule UpdateFunc {"managedCluster": "local-cluster"} 1.6913935142303293e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "local-cluster"} 1.691393514230412e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935142304192e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.691393514230472e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935142304778e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935142304814e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.691393514230485e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935142304883e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935142304916e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935142304952e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935142309484e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935142310636e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "local-cluster", "request.name": "local-cluster", "request.namespace": ""} 1.6913935142310808e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "local-cluster"} 1.6913935142310956e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935142311075e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935142311208e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "local-cluster"} 1.6913935142395153e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "local-cluster"} 1.6913935143059847e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "acm-interops-opp-ci-gcp-1691390716081", "Request.Name": "observability-addon"} 1.6913935143060932e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935143060987e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935143061676e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935143061738e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935143061774e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.691393514306181e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935143061843e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935143061876e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.691393514306203e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935143067482e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935143068979e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "observability-addon", "request.namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143069146e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143069444e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143069563e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143069801e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143096359e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143209236e+09 INFO controller_placementrule Updated status for managedclusteraddon {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143479872e+09 INFO controller_placementrule UpdateFunc {"managedCluster": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143480957e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143482127e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935143482237e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935143482807e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935143482916e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935143482964e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935143483007e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935143483052e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935143483095e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.691393514348314e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.691393514348815e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935143489594e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "acm-interops-opp-ci-gcp-1691390716081", "request.namespace": ""} 1.6913935143489969e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143490255e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143490548e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143490818e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935143505306e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935144218152e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935144267867e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935144317904e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935144366677e+09 INFO deploying Update {"Kind:": "apps/v1, Kind=StatefulSet", "Name:": "observability-alertmanager"} 1.691393514442106e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "acm-interops-opp-ci-gcp-1691390716081", "Request.Name": "observability-addon"} 1.691393514442296e+09 INFO util observability-controller clustermanagementaddon is present 1.691393514442304e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935144424038e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.691393514442413e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.691393514442459e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935144424648e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935144424696e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.691393514442474e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935144424775e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935144431384e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935144432604e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "observability-addon", "request.namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.69139351444329e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935144433036e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935144433162e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935144433293e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935144457781e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935144558296e+09 INFO controller_placementrule Updated status for managedclusteraddon {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935144968977e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393514503543e+09 INFO deploying Update {"Kind:": "/v1, Kind=ConfigMap", "Name:": "thanos-ruler-config"} 1.691393514529074e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "acm-interops-opp-ci-gcp-1691390716081", "Request.Name": "observability-addon"} 1.6913935145291626e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935145291677e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935145292256e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.691393514529231e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.691393514529235e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935145292385e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.691393514529242e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935145292459e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935145292492e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935145297122e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935145298223e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "observability-addon", "request.namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514529837e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935145298498e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935145298617e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514529877e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514533368e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935145387893e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935145408695e+09 INFO deploying Update {"Kind:": "apps/v1, Kind=Deployment", "Name:": "observability-rbac-query-proxy"} 1.6913935145611057e+09 INFO controller_placementrule Updated status for managedclusteraddon {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393514565987e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935145776827e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.69139351462414e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.691393514624227e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935146242566e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935146243358e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935146243756e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935146244009e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935146244998e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935146245494e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935146246564e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935146246824e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935146246881e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935146246934e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935146247072e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935146247115e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.691393514624726e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935146247349e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935146260643e+09 INFO controller_multiclustermonitoring Updating observatorium CR {"observatorium": "observability"} 1.6913935146543276e+09 INFO controller_multiclustermonitoring Updating grafana datasource secret 1.6913935146634645e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935146777604e+09 INFO util Update deployment restart label {"name": "observability-grafana"} 1.691393514677809e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935146778452e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935146781983e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.691393514678207e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.691393514678215e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935146782198e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393514678692e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935146787012e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935146787057e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.691393514678715e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935146787188e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935146787226e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935146793096e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.691393514679322e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935146793275e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393514679337e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935146793416e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935146795266e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935146795342e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935146801898e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935147087524e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935147133925e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935147216332e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393514732397e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935147585795e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.69139351476974e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.691393515541711e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935155447116e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935155477138e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935156518197e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393515703774e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935157119067e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935157433019e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935157594519e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935157594702e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.691393515759475e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935157595298e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935157595499e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.691393515759556e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935157596533e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935157596743e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935157597334e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.691393515759746e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935157597518e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.691393515759758e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935157597878e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935157597995e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935157598028e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935157598076e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935157613785e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935157614245e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935157614272e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935157617588e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.691393515761772e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.69139351576178e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935157617848e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935157621634e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935157621758e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.691393515762181e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.69139351576219e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935157621944e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935157621984e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393515762664e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935157626765e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935157626827e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393515762693e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.691393515762698e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393515762841e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935157628536e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935157720952e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935157748883e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935157776818e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935157843893e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.691393515845572e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393515848879e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935171403172e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935171432433e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935171471138e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935172502158e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935173023458e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935173138592e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.691393517337728e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935173537762e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935173537924e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935173537967e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935173538394e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935173538506e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935173538551e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393517353914e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935173539333e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.691393517353951e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935173539639e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935173539686e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935173539743e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935173539855e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935173539891e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935173539932e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935173539994e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935173551319e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935173551722e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935173552668e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935173554363e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935173554478e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935173554566e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.691393517355461e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935173557882e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935173558e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935173558054e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.691393517355815e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935173558197e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935173558238e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393517356211e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935173562226e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935173562286e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935173562384e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935173562427e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935173563876e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935173563983e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935173657415e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935173688176e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935173720677e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935173843977e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935174440343e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935174468644e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935187780704e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393518800522e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935188198156e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.691393518974711e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935189968202e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "acm-interops-opp-ci-gcp-1691390716081", "Request.Name": "observability-addon"} 1.691393518996938e+09 INFO util observability-controller clustermanagementaddon is present 1.691393518996946e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935189970264e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935189970393e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.691393518997044e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935189970481e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935189970558e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935189970593e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.691393518997063e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935189975586e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935189976907e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "observability-addon", "request.namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935189977133e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935189977295e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935189977403e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393518997752e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190000432e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190200386e+09 INFO controller_placementrule Updated status for managedclusteraddon {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190514514e+09 INFO controller_placementrule UpdateFunc {"managedCluster": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190515387e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190542717e+09 INFO util observability-controller clustermanagementaddon is present 1.691393519054283e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.6913935190543365e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.691393519054342e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.691393519054346e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935190543497e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935190543532e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935190543568e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935190543602e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935190548258e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935190549262e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "acm-interops-opp-ci-gcp-1691390716081", "request.namespace": ""} 1.6913935190549417e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393519054957e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190549684e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190549822e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935190624557e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.691393519072854e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935191209695e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935192275848e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393519298396e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935192984269e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935192984333e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935192985039e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935192985127e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935192985163e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393519298592e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.69139351929862e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935192986712e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935192986822e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935192986858e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935192986903e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935192987006e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935192987041e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935192987077e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935192987137e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935193004673e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935193004925e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935208540106e+09 INFO controller_placementrule UpdateFunc {"managedCluster": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935208540995e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935208542335e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935208542426e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.691393520854322e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.691393520854329e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935208543332e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935208543372e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.691393520854342e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.691393520854346e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.69139352085435e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935208551064e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935208552306e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "acm-interops-opp-ci-gcp-1691390716081", "request.name": "acm-interops-opp-ci-gcp-1691390716081", "request.namespace": ""} 1.691393520855246e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935208552608e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935208552723e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935208552866e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935208623338e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "acm-interops-opp-ci-gcp-1691390716081"} 1.6913935245365915e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935245371108e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935245371501e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935245371776e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.691393524537197e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935245376596e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935245376735e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935245376792e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935245376906e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935245376952e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935245377e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935245382566e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935245382729e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393524538279e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935245382905e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935245382953e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935245384405e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.691393524538453e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935245526636e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935245559404e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935245591755e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935245944586e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935246447372e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393524662305e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935250081108e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393525011653e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935250153542e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.691393525076423e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935251223137e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.691393525132219e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935251410038e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935251591136e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935251591408e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935251591468e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935251592524e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.691393525159293e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935251593025e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935251594217e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935251594458e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935251595063e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935251595225e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935251595278e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.691393525159533e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935251595445e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935251595488e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935251595528e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935251595588e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935251613073e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.691393525161356e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.691393525161779e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935251618056e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935251618135e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935251618178e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935251623023e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935251623154e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935251623204e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.69139352516233e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.691393525162353e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.691393525162358e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935251630707e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.691393525163085e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393525163106e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935251631186e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935251631231e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935251633513e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935251633632e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935251640487e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935251748707e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935251791131e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.691393525182991e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.69139352519091e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935252072978e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935252105713e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935261196477e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935261240964e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.691393526127896e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935262680593e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935263344443e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935263726964e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935264023995e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935265370786e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935265371797e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935265371952e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935265372624e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935265372796e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935265372844e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935265373743e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.691393526537398e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.691393526537464e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935265374866e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935265374966e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935265375037e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.691393526537516e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935265375202e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935265375245e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935265375314e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935265390532e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.691393526539104e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.691393526539174e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935265393977e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.69139352653941e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935265394175e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.691393526539422e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393526539793e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935265398061e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935265398095e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935265398161e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935265398188e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.691393526539821e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393526540257e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.69139352654027e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935265402737e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393526540281e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.691393526540287e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935265404363e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935265404468e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.691393526595987e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935266107128e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.691393526615817e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935266388366e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935266757915e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935266803098e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935277689476e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935277894344e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935278000839e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935279169397e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935279716332e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935279818249e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935279943042e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935280243578e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935280243926e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935280243993e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393528024463e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935280244744e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935280244782e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393528024556e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.691393528024581e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.691393528024666e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935280246792e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935280246832e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935280246882e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935280246994e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935280247028e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935280247066e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935280247133e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935280264432e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935280264962e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935280265644e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935280268686e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.691393528026882e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.69139352802689e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.691393528026894e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935280273204e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935280273323e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935280273373e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935280273466e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.691393528027351e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935280273545e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935280279398e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935280279546e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.69139352802796e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935280279708e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935280279748e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935280282042e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.691393528028211e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.69139352807064e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935280803823e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935280946763e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935281062655e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.691393528133127e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935281368914e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935293219829e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935293296769e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.691393529337811e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935294694889e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935295430264e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935295517225e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935295643249e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935295843222e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935295843475e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935295843532e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935295844104e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935295844314e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935295844364e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935295845106e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935295845315e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935295845883e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935295846019e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935295846167e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935295846226e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.691393529584634e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935295846374e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.691393529584641e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935295846472e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.69139352958634e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935295863888e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.691393529586751e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935295867662e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935295867739e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935295867782e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935295872338e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935295872445e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935295872507e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935295872612e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935295872653e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935295872693e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393529587848e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935295878642e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935295878694e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935295878804e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935295878851e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935295880728e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935295880868e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935295887082e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935295994983e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935296044312e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935296075351e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935296188974e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935296357968e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393529639498e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935297404954e+09 INFO controller_placementrule UpdateFunc {"managedCluster": "local-cluster"} 1.6913935297406604e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "", "Request.Name": "local-cluster"} 1.6913935297407568e+09 INFO util observability-controller clustermanagementaddon is present 1.691393529740762e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.691393529740862e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.6913935297408702e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935297408748e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935297408788e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935297408824e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935297408876e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.691393529740891e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935297414596e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935297416866e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "local-cluster", "request.name": "local-cluster", "request.namespace": ""} 1.6913935297417045e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "local-cluster"} 1.691393529741718e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935297417297e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935297417436e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "local-cluster"} 1.6913935297436492e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "local-cluster"} 1.691393530919826e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393530923172e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935309261768e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935310286312e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935310738869e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935310811973e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935311127768e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935311288385e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935311288657e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935311288738e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935311289337e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935311289554e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935311289613e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935311290288e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935311290503e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935311291142e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935311291296e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.69139353112914e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935311291473e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.691393531129162e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935311291666e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935311291714e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935311291792e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935311307855e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.691393531130832e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.6913935311309144e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.691393531131147e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935311311593e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.691393531131167e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935311311712e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935311316211e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935311316357e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935311316411e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935311316512e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935311316555e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935311316595e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393531132172e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935311321845e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935311321893e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935311321983e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.691393531132202e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935311323576e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.691393531132366e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935311445727e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935311483264e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935311513178e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393531158853e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935312216396e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935312248907e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.69139353187225e+09 INFO controller_placementrule Reconciling PlacementRule {"Request.Namespace": "local-cluster", "Request.Name": "observability-addon"} 1.6913935318724115e+09 INFO util observability-controller clustermanagementaddon is present 1.6913935318724456e+09 INFO controller_placementrule load and update templates for managedcluster resources 1.691393531872537e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-config-reloader-rhel8@sha256:1339ca942f881dcbd3a4fb0d02226729d50025bc8728a2cb9fae40493f6470e7"} 1.6913935318725653e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-prometheus-rhel8@sha256:d66bcaaa6d600712b213a58bd0908a6f9ddad4c39c1bd6ef9a4d4f24219b9690"} 1.6913935318725865e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-rbac-proxy-rhel8@sha256:54a22975663671739131c6f9f2d2b070562a66607ff7bcf31404a85d34a667ad"} 1.6913935318726513e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/kube-state-metrics-rhel8@sha256:dd5a7ba899501ca68e0e3b090aa6ddd5cd0ae977451e85adedcadeb13e974253"} 1.69139353187268e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/metrics-collector-rhel8@sha256:6960d300cf5d6adce2f3baa73badf7bc71da868cbefaee7f6987ecd7c01979f4"} 1.6913935318727033e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/node-exporter-rhel8@sha256:99e61049f49c7343e884e4177b8305cde32f8f2009b1d86a58b9b7567e4cae3a"} 1.6913935318727217e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-rhel8@sha256:6d46e65dce74fe4ecb02b01745a9646759440a93484f97d5085babbfb6d569c1"} 1.6913935318732274e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/endpoint-monitoring-rhel8-operator@sha256:874e76e2c0bb75443dc76bde96fcd8b21eb6dd7cd1847e584c7daab724a21c1b"} 1.6913935318733664e+09 INFO controller_placementrule Monitoring operator should be installed in cluster {"cluster_name": "local-cluster", "request.name": "observability-addon", "request.namespace": "local-cluster"} 1.691393531873405e+09 INFO controller_placementrule observabilityaddon already existed/unchanged {"namespace": "local-cluster"} 1.691393531873438e+09 INFO controller_placementrule clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935318734674e+09 INFO controller_placementrule rolebinding endpoint-observability-res-rolebinding already existed/unchanged {"namespace": "local-cluster"} 1.6913935318737016e+09 INFO util ManagedClusterAddOn is created or updated successfully {"name": "observability-controller", "namespace": "local-cluster"} 1.6913935318755777e+09 INFO controller_placementrule manifestwork already existed/unchanged {"namespace": "local-cluster"} 1.6913935318855646e+09 INFO controller_placementrule Updated status for managedclusteraddon {"namespace": "local-cluster"} 1.6913935325160458e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935325208244e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935325259478e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.69139353262415e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935326741054e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935326819227e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935327114124e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935327290564e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935327290773e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935327290812e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935327291396e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935327291565e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935327291615e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935327292395e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935327292724e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935327293253e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935327293367e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.69139353272934e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935327293458e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.691393532729357e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935327293618e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935327293658e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935327293723e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935327308972e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935327309322e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.691393533435241e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.691393533435587e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.6913935334356e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935334356184e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935334356241e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393533435996e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935334360075e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935334360108e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935334360182e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935334360204e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935334360225e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935334364982e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935334365098e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935334365134e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935334365203e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935334365227e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935334367037e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.6913935334367523e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.6913935334484887e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935334527895e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935334559262e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935334730942e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935334942884e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935334976604e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.6913935341172128e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935341201572e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935341229928e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935342259786e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935342751467e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935342838662e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.69139353431086e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393534327598e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935343276482e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935343276548e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935343277173e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935343277361e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.691393534327741e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.69139353432782e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935343278491e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.691393534327899e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935343279119e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.6913935343279164e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935343279216e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935343279338e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.691393534327938e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935343279424e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.6913935343279488e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393534329222e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.691393534329258e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. 1.6913935544505618e+09 INFO controller_multiclustermonitoring Reconciling MultiClusterObservability {"Request.Namespace": "", "Request.Name": "observability"} 1.691393554451345e+09 DEBUG config operand is {"key": "grafana", "name": "observability-grafana"} 1.691393554451361e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/acm-grafana-rhel8@sha256:a8e6511d31096b348573d6877321b4ecca2bd221f6144d049c0c9a8e1e3493e9"} 1.6913935544513698e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/grafana-dashboard-loader-rhel8@sha256:2de872fc1ec72b7da02f92627e786fe16a0e17a608422e685ba61a838b6db480"} 1.6913935544513738e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393554451771e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935544517844e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935544517877e+09 DEBUG config operand is {"key": "alertmanager", "name": "observability-alertmanager"} 1.6913935544517946e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/prometheus-alertmanager-rhel8@sha256:424b0eab6cdfee1ba77674ecaed24759c433eb7b46b563e156083905ed77f3c0"} 1.6913935544517975e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935544517996e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.691393554452241e+09 DEBUG config operand is {"key": "rbac-query-proxy", "name": "observability-rbac-query-proxy"} 1.6913935544522521e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935544522555e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935544522617e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/rbac-query-proxy-rhel8@sha256:5c13359597bad0df4fc3e3d81f64aea4e22cc8479b28c5d6bd3eaf57f60ae9a9"} 1.6913935544522648e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:41762eda03298362d3f97e1fd0be4af429d22658b520f94023941655e062d1a1"} 1.6913935544523954e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8-operator@sha256:5247ee151ff046c0e1bdeb2a92daac45048f57b4150cbf9550ecd15742842deb"} 1.691393554452404e+09 DEBUG config operand is {"key": "observatorium-operator", "name": "observability-observatorium-operator"} 1.691393554463376e+09 INFO deploying deployerFn not found {"kind:": "Role"} 1.6913935544664972e+09 INFO deploying deployerFn not found {"kind:": "RoleBinding"} 1.6913935544706955e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935544779058e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=ConfigMap", "Name:": "observability-managed-cluster-label-allowlist"} 1.6913935544922101e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935544951422e+09 INFO deploying Update {"Kind:": "/v1, Kind=Secret", "Name:": "grafana-config"} 1.691393554654731e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935546578112e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-config"} 1.6913935546613584e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "alertmanager-proxy"} 1.6913935546947012e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.6913935547158904e+09 INFO deploying Skip creation {"Kind:": "/v1, Kind=Secret", "Name:": "rbac-proxy-cookie-secret"} 1.6913935547239451e+09 INFO deploying deployerFn not found {"kind:": "Ingress"} 1.6913935547308657e+09 INFO deploying deployerFn not found {"kind:": "ServiceAccount"} 1.691393554746909e+09 INFO controller_certificates CA secrets already existed {"name": "observability-server-ca-certs"} 1.6913935547469363e+09 INFO controller_certificates CA secrets already existed {"name": "observability-client-ca-certs"} 1.6913935547469437e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935547470057e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-server-certs"} 1.6913935547470295e+09 INFO controller_certificates Certificate secrets already existed {"name": "observability-grafana-certs"} 1.6913935547470355e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.691393554747097e+09 INFO controller_multiclustermonitoring storageClassSelected {"storageClassSelected": "gp3-csi"} 1.6913935547471175e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/observatorium-rhel8@sha256:318264913ab5b742be1fee4d1bdb03fa52430df5f475801e30401b2ca83c4964"} 1.6913935547471676e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/openshift4/ose-configmap-reloader@sha256:fa5b2e42a27e3ab5f0a0d2a103b5602b81d1b29eb788b54cc60ad547744ad053"} 1.6913935547471807e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.691393554747189e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935547471945e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-receive-controller-rhel8@sha256:ef6bf8a0ddcd544014b1878aa48bb0297bce93544ffb049cbc627121166f8da2"} 1.6913935547472079e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-rhel8@sha256:736c816f826fecabb3101be7ca7ce41b755e8ea8321907782904ccacfbbbd9ec"} 1.691393554747212e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/memcached-exporter-rhel8@sha256:c1c6f450be6b17230a7f811642417ff843f044101b6fedabb57b6a81d7108fa0"} 1.6913935547472162e+09 DEBUG config image replacement {"componentName": "registry.redhat.io/rhacm2/thanos-rhel8@sha256:bb8d79454abb609f866c3385d698d0a7128188ea84e049499d913a942ceb9031"} 1.691393554747222e+09 DEBUG config operand is {"key": "observatorium", "name": "observability"} 1.6913935547483432e+09 INFO controller_multiclustermonitoring StorageVersionMigration already existed/unchanged {"name": "storage-version-migration-observability"} 1.6913935547483764e+09 DEBUG controller_multiclustermonitoring status update goroutine is triggered. Failure [1661.895 seconds] [BeforeSuite] BeforeSuite  /tmp/obs/tests/pkg/tests/observability-e2e-test_suite_test.go:132 Timed out after 1500.000s. Expected success, but got an error: <*errors.errorString | 0xc000fea550>: { s: "statefulset observability-thanos-receive-default should have 2 but got 1 ready replicas", } statefulset observability-thanos-receive-default should have 2 but got 1 ready replicas /tmp/obs/tests/pkg/tests/observability_install_test.go:229 ------------------------------ I0807 07:57:54.078055 7951 utils.go:58] Create kubeclient for url https://api.ci-op-id52jm5k-51b43.cspilp.interop.ccitredhat.com:6443 using kubeconfig path /alabama/.kube/config I0807 07:57:54.078068 7951 utils.go:211] Kubeconfig path /alabama/.kube/config I0807 07:57:54.113119 7951 mco_deploy.go:161] Get <31> pods in namespace JUnit report was created: /tmp/obs/tests/pkg/tests/junit_results.xml Ran 62 of 0 Specs in 1661.931 seconds FAIL! -- 0 Passed | 62 Failed | 0 Pending | 0 Skipped --- FAIL: TestObservabilityE2E (1661.94s) FAIL You're using deprecated Ginkgo functionality: ============================================= Ginkgo 2.0 is under active development and will introduce (a small number of) breaking changes. To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/v2/docs/MIGRATING_TO_V2.md To comment, chime in at https://github.com/onsi/ginkgo/issues/711 You are using a custom reporter. Support for custom reporters will likely be removed in V2. Most users were using them to generate junit or teamcity reports and this functionality will be merged into the core reporter. In addition, Ginkgo 2.0 will support emitting a JSON-formatted report that users can then manipulate to generate custom reports. If this change will be impactful to you please leave a comment on https://github.com/onsi/ginkgo/issues/711 Learn more at: https://github.com/onsi/ginkgo/blob/v2/docs/MIGRATING_TO_V2.md#removed-custom-reporters To silence deprecations that can be silenced set the following environment variable: ACK_GINKGO_DEPRECATIONS=1.16.4 Ginkgo ran 1 suite in 27m42.586160231s Test Suite Failed