Name: thanos-querier-7d86b7fcb6-8zm26 Namespace: openshift-monitoring Priority: 2000000000 Priority Class Name: system-cluster-critical Node: apzhnosni01.internal.six-group.net/10.147.47.7 Start Time: Thu, 08 Sep 2022 13:43:07 +0200 Labels: app.kubernetes.io/component=query-layer app.kubernetes.io/instance=thanos-querier app.kubernetes.io/managed-by=cluster-monitoring-operator app.kubernetes.io/name=thanos-query app.kubernetes.io/part-of=openshift-monitoring app.kubernetes.io/version=0.23.1 pod-template-hash=7d86b7fcb6 Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "172.18.0.142" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "172.18.0.142" ], "default": true, "dns": {} }] openshift.io/scc: restricted Status: Running IP: 172.18.0.142 IPs: IP: 172.18.0.142 Controlled By: ReplicaSet/thanos-querier-7d86b7fcb6 Containers: thanos-query: Container ID: cri-o://3457595bbb7a879ab82a18834fcc9af06d277e0215a8ccd8542e334fdc327fa7 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:037a178d5e046fdae2d51a7e758713adcb57348471fde203e0bd3af4c6f7b77b Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:037a178d5e046fdae2d51a7e758713adcb57348471fde203e0bd3af4c6f7b77b Port: 9090/TCP Host Port: 0/TCP Args: query --grpc-address=127.0.0.1:10901 --http-address=127.0.0.1:9090 --log.format=logfmt --query.replica-label=prometheus_replica --query.replica-label=thanos_ruler_replica --store=dnssrv+_grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local --query.auto-downsampling --store.sd-dns-resolver=miekgdns --grpc-client-tls-secure --grpc-client-tls-cert=/etc/tls/grpc/client.crt --grpc-client-tls-key=/etc/tls/grpc/client.key --grpc-client-tls-ca=/etc/tls/grpc/ca.crt --grpc-client-server-name=prometheus-grpc --rule=dnssrv+_grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local --target=dnssrv+_grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local --store=dnssrv+_grpc._tcp.prometheus-operated.openshift-user-workload-monitoring.svc.cluster.local --store=dnssrv+_grpc._tcp.thanos-ruler-operated.openshift-user-workload-monitoring.svc.cluster.local --rule=dnssrv+_grpc._tcp.prometheus-operated.openshift-user-workload-monitoring.svc.cluster.local --rule=dnssrv+_grpc._tcp.thanos-ruler-operated.openshift-user-workload-monitoring.svc.cluster.local --target=dnssrv+_grpc._tcp.prometheus-operated.openshift-user-workload-monitoring.svc.cluster.local --log.level=debug --request.logging-config=http: options: level: DEBUG decision: log_start: false log_end: true grpc: options: level: DEBUG decision: log_start: false log_end: true State: Running Started: Mon, 12 Sep 2022 17:16:18 +0200 Last State: Terminated Reason: Error Message: 2:10901 LabelSets: {prometheus=\"openshift-monitoring/k8s\", prometheus_replica=\"prometheus-k8s-1\", six_alarm_type=\"AUTO_OM\", six_alerting_email=\"helpdesk-container@six-group.com\", six_application_ids=\"A0643\", six_cluster=\"Napoli\", six_cluster_short=\"NAP01\", six_incident=\"False\", six_resp=\"2020\", six_resp_type=\"PHONE\", six_support=\"5x8\"} Mint: -62167219200000 Maxt: 9223372036854775807 queried" ts=2022-09-12T15:16:17.757767326Z caller=level.go:63 protocol=http http.component=server http.method="GET /api/v1/query_range?start=1662992177.723&end=1662995777.723&step=60&query=%0A++++++sum%28%0A++++++++instance%3Anode_network_receive_bytes_excluding_lo%3Arate1m%0A++++++++*%0A++++++++on%28instance%29+group_left%28role%29+%28%0A++++++++++label_replace%28max+by+%28node%29+%28kube_node_role%7Brole%3D%7E%22.%2B%22%7D%29%2C+%22instance%22%2C+%22%241%22%2C+%22node%22%2C+%22%28.*%29%22%29%0A++++++++%29%0A++++++%29%0A++++" http.request_id=01GCS4BJ6MXF482Q7P4NC2112A http.status_code=200 http.time_ms=9.101 http.remote_addr=127.0.0.1:52044 thanos.method_name=query_range level=debug msg="finished call" panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1214084] goroutine 2994807 [running]: github.com/thanos-io/thanos/pkg/rules.(*rulesServer).Send(0xc001f02550, 0xc00613a120) /go/src/github.com/improbable-eng/thanos/pkg/rules/rules.go:160 +0x144 github.com/thanos-io/thanos/pkg/rules.(*rulesStream).receive(0xc0017a2b70, {0x1e0d388, 0xc003ad6880}) /go/src/github.com/improbable-eng/thanos/pkg/rules/proxy.go:136 +0x684 github.com/thanos-io/thanos/pkg/rules.(*Proxy).Rules.func1() /go/src/github.com/improbable-eng/thanos/pkg/rules/proxy.go:60 +0x25 golang.org/x/sync/errgroup.(*Group).Go.func1() /go/src/github.com/improbable-eng/thanos/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x67 created by golang.org/x/sync/errgroup.(*Group).Go /go/src/github.com/improbable-eng/thanos/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x92 Exit Code: 2 Started: Mon, 12 Sep 2022 12:20:47 +0200 Finished: Mon, 12 Sep 2022 17:16:17 +0200 Ready: True Restart Count: 3 Requests: cpu: 10m memory: 12Mi Environment: HOST_IP_ADDRESS: (v1:status.hostIP) Mounts: /etc/tls/grpc from secret-grpc-tls (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2hg7 (ro) oauth-proxy: Container ID: cri-o://7544f3b8d7832f6c2e1520844fd73c718658b54c948b0bf7ee5765845056e287 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dcb61fe5e82c0d01b90155f20b0b6ce37446867ca0d005b727fc073f86ed1354 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dcb61fe5e82c0d01b90155f20b0b6ce37446867ca0d005b727fc073f86ed1354 Port: 9091/TCP Host Port: 0/TCP Args: -provider=openshift -https-address=:9091 -http-address= -email-domain=* -upstream=http://localhost:9090 -openshift-service-account=thanos-querier -openshift-sar={"resource": "namespaces", "verb": "get"} -openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}} -tls-cert=/etc/tls/private/tls.crt -tls-key=/etc/tls/private/tls.key -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token -cookie-secret-file=/etc/proxy/secrets/session_secret -openshift-ca=/etc/pki/tls/cert.pem -openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt -bypass-auth-for=^/-/(healthy|ready)$ -htpasswd-file=/etc/proxy/htpasswd/auth State: Running Started: Thu, 08 Sep 2022 13:43:10 +0200 Ready: True Restart Count: 0 Requests: cpu: 1m memory: 20Mi Liveness: http-get https://:9091/-/healthy delay=5s timeout=1s period=30s #success=1 #failure=4 Readiness: http-get https://:9091/-/ready delay=5s timeout=1s period=5s #success=1 #failure=20 Environment: HTTP_PROXY: HTTPS_PROXY: NO_PROXY: Mounts: /etc/pki/ca-trust/extracted/pem/ from thanos-querier-trusted-ca-bundle (ro) /etc/proxy/htpasswd from secret-thanos-querier-oauth-htpasswd (rw) /etc/proxy/secrets from secret-thanos-querier-oauth-cookie (rw) /etc/tls/private from secret-thanos-querier-tls (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2hg7 (ro) kube-rbac-proxy: Container ID: cri-o://3d636fe20107db2d365e1223d91da619ea3e650308dd321086a3fe966cffde68 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d486bed1127bd9cb55d0ca6ce3ecaad92fb25d5aee58103744e6bf90f83243a9 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d486bed1127bd9cb55d0ca6ce3ecaad92fb25d5aee58103744e6bf90f83243a9 Port: 9092/TCP Host Port: 0/TCP Args: --secure-listen-address=0.0.0.0:9092 --upstream=http://127.0.0.1:9095 --config-file=/etc/kube-rbac-proxy/config.yaml --tls-cert-file=/etc/tls/private/tls.crt --tls-private-key-file=/etc/tls/private/tls.key --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --logtostderr=true --allow-paths=/api/v1/query,/api/v1/query_range,/api/v1/labels,/api/v1/label/*/values,/api/v1/series --tls-min-version=VersionTLS12 State: Running Started: Thu, 08 Sep 2022 13:43:10 +0200 Ready: True Restart Count: 0 Requests: cpu: 1m memory: 15Mi Environment: Mounts: /etc/kube-rbac-proxy from secret-thanos-querier-kube-rbac-proxy (rw) /etc/tls/private from secret-thanos-querier-tls (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2hg7 (ro) prom-label-proxy: Container ID: cri-o://83f16a7e673b96f602793391504ef89801e44d1e7636983e4ac1c751756d4dbe Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9053faaf26e77ee41b159cc91af3bfdc9e76c5d707e5b4675b462d1be45af267 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9053faaf26e77ee41b159cc91af3bfdc9e76c5d707e5b4675b462d1be45af267 Port: Host Port: Args: --insecure-listen-address=127.0.0.1:9095 --upstream=http://127.0.0.1:9090 --label=namespace --enable-label-apis --error-on-replace State: Running Started: Thu, 08 Sep 2022 13:43:10 +0200 Ready: True Restart Count: 0 Requests: cpu: 1m memory: 15Mi Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2hg7 (ro) kube-rbac-proxy-rules: Container ID: cri-o://ba2863ec96d0eaa4ff691e7dee47f9eaf3f180d9658fcd46260713667c8ec2d0 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d486bed1127bd9cb55d0ca6ce3ecaad92fb25d5aee58103744e6bf90f83243a9 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d486bed1127bd9cb55d0ca6ce3ecaad92fb25d5aee58103744e6bf90f83243a9 Port: 9093/TCP Host Port: 0/TCP Args: --secure-listen-address=0.0.0.0:9093 --upstream=http://127.0.0.1:9095 --config-file=/etc/kube-rbac-proxy/config.yaml --tls-cert-file=/etc/tls/private/tls.crt --tls-private-key-file=/etc/tls/private/tls.key --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --logtostderr=true --allow-paths=/api/v1/rules --tls-min-version=VersionTLS12 State: Running Started: Thu, 08 Sep 2022 13:43:10 +0200 Ready: True Restart Count: 0 Requests: cpu: 1m memory: 15Mi Environment: Mounts: /etc/kube-rbac-proxy from secret-thanos-querier-kube-rbac-proxy-rules (rw) /etc/tls/private from secret-thanos-querier-tls (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2hg7 (ro) kube-rbac-proxy-metrics: Container ID: cri-o://fb13147d72c82b0286f77ce0d3de3be9c1e839284e898ab33de6e237a32703a1 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d486bed1127bd9cb55d0ca6ce3ecaad92fb25d5aee58103744e6bf90f83243a9 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d486bed1127bd9cb55d0ca6ce3ecaad92fb25d5aee58103744e6bf90f83243a9 Port: 9094/TCP Host Port: 0/TCP Args: --secure-listen-address=0.0.0.0:9094 --upstream=http://127.0.0.1:9090 --config-file=/etc/kube-rbac-proxy/config.yaml --tls-cert-file=/etc/tls/private/tls.crt --tls-private-key-file=/etc/tls/private/tls.key --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --client-ca-file=/etc/tls/client/client-ca.crt --logtostderr=true --allow-paths=/metrics --tls-min-version=VersionTLS12 State: Running Started: Thu, 08 Sep 2022 13:43:11 +0200 Ready: True Restart Count: 0 Requests: cpu: 1m memory: 15Mi Environment: Mounts: /etc/kube-rbac-proxy from secret-thanos-querier-kube-rbac-proxy-metrics (rw) /etc/tls/client from metrics-client-ca (ro) /etc/tls/private from secret-thanos-querier-tls (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2hg7 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: secret-thanos-querier-tls: Type: Secret (a volume populated by a Secret) SecretName: thanos-querier-tls Optional: false secret-thanos-querier-oauth-cookie: Type: Secret (a volume populated by a Secret) SecretName: thanos-querier-oauth-cookie Optional: false secret-thanos-querier-kube-rbac-proxy: Type: Secret (a volume populated by a Secret) SecretName: thanos-querier-kube-rbac-proxy Optional: false secret-thanos-querier-kube-rbac-proxy-rules: Type: Secret (a volume populated by a Secret) SecretName: thanos-querier-kube-rbac-proxy-rules Optional: false secret-thanos-querier-kube-rbac-proxy-metrics: Type: Secret (a volume populated by a Secret) SecretName: thanos-querier-kube-rbac-proxy-metrics Optional: false metrics-client-ca: Type: ConfigMap (a volume populated by a ConfigMap) Name: metrics-client-ca Optional: false thanos-querier-trusted-ca-bundle: Type: ConfigMap (a volume populated by a ConfigMap) Name: thanos-querier-trusted-ca-bundle-53usmrh3q2872 Optional: true secret-thanos-querier-oauth-htpasswd: Type: Secret (a volume populated by a Secret) SecretName: thanos-querier-oauth-htpasswd Optional: false secret-grpc-tls: Type: Secret (a volume populated by a Secret) SecretName: thanos-querier-grpc-tls-7sg58biqpl5vi Optional: false kube-api-access-c2hg7: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/infra= Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 3m38s (x4 over 4d3h) kubelet Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:037a178d5e046fdae2d51a7e758713adcb57348471fde203e0bd3af4c6f7b77b" already present on machine Normal Created 3m38s (x4 over 4d3h) kubelet Created container thanos-query Normal Started 3m38s (x4 over 4d3h) kubelet Started container thanos-query