Loading...

Type: Bug
Resolution: Not a Bug
Priority: Undefined
Fix Version/s: None
Affects Version/s: ACM 2.13.0
Component/s: Observability
Labels:
None

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Intelligence Requested:
Market:

Severity:
Important

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem:

http_proxy 4.18.4 cluster, install ACM and ACM observability, login grafana UI, 500 error in grafana-proxy container logs, checked the grafana deploy and pod file, no proxy seting under grafana-proxy container

$ oc get proxy/cluster -oyaml
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
  creationTimestamp: "2025-03-14T07:22:28Z"
  generation: 1
  name: cluster
  resourceVersion: "449"
  uid: 193b53e0-a42b-48c6-8ba5-a49c7e69166b
spec:
  httpProxy: http://proxy-user2:***@***:3128
  httpsProxy: http://proxy-user2:***@***:3128
  noProxy: test.no-proxy.com
  trustedCA:
    name: ""
status:
  httpProxy: http://proxy-user2:***@***:3128
  httpsProxy: http://proxy-user2:***@***:3128
  noProxy: .cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.**.qe.devcluster.openshift.com,localhost,test.no-proxy.com

500 error in grafana-proxy container logs

$ oc -n open-cluster-management-observability get pod | grep grafana
observability-grafana-797958984f-8wlzt                     3/3     Running   0             28m
observability-grafana-797958984f-n7qpl                     3/3     Running   0             28m
observability-grafana-797958984f-ns4pj                     3/3     Running   0             28m

$  oc -n open-cluster-management-observability get route | grep grafana
grafana             grafana-open-cluster-management-observability.apps.juzhao-proxy4.qe.devcluster.openshift.com                       grafana                           oauth-proxy   reencrypt/Redirect   None

$  oc -n open-cluster-management-observability logs -c grafana-proxy  observability-grafana-797958984f-n7qpl
...
2025/03/14 14:26:45 oauthproxy.go:661: error redeeming code (client:10.131.0.73:33034): Post "https://oauth-openshift.apps.juzhao-proxy4.qe.devcluster.openshift.com/oauth/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2025/03/14 14:26:45 oauthproxy.go:452: ErrorPage 500 Internal Error Internal Error
2025/03/14 14:31:46 provider.go:631: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2025/03/14 14:31:46 provider.go:671: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.juzhao-proxy4.qe.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.juzhao-proxy4.qe.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.juzhao-proxy4.qe.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2025/03/14 14:32:46 oauthproxy.go:661: error redeeming code (client:10.131.0.73:48860): Post "https://oauth-openshift.apps.juzhao-proxy4.qe.devcluster.openshift.com/oauth/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

checked the grafana deploy and pod file, no proxy setting injested to grafana-proxy container, see grafana deploy file

$ oc -n open-cluster-management-observability get deploy observability-grafana -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2025-03-14T14:08:53Z"
  generation: 1
  labels:
    app: multicluster-observability-grafana
    observability.open-cluster-management.io/name: observability
  name: observability-grafana
  namespace: open-cluster-management-observability
  ownerReferences:
  - apiVersion: observability.open-cluster-management.io/v1beta2
    blockOwnerDeletion: true
    controller: true
    kind: MultiClusterObservability
    name: observability
    uid: 1fbf9f96-d1aa-4086-87ae-6b3707817b54
  resourceVersion: "228629"
  uid: 246dea3f-290a-470e-b5fd-bfc3168d4996
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: multicluster-observability-grafana
      observability.open-cluster-management.io/name: observability
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: multicluster-observability-grafana
        observability.open-cluster-management.io/name: observability
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - multicluster-observability-grafana
              topologyKey: topology.kubernetes.io/zone
            weight: 70
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - multicluster-observability-grafana
              topologyKey: kubernetes.io/hostname
            weight: 30
      containers:
      - args:
        - -config=/etc/grafana/grafana.ini
        env:
        - name: SQLITE_TMPDIR
          value: /var/lib/grafana
        image: quay.io:443/acm-d/acm-grafana-rhel9@sha256:46404fb6813c88e6f51725c173631ede49106590b58fca2d59ad4f2b1a57a427
        imagePullPolicy: IfNotPresent
        name: grafana
        ports:
        - containerPort: 3001
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: 4m
            memory: 100Mi
        securityContext:
          privileged: false
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/grafana
          name: grafana-storage
        - mountPath: /etc/grafana/provisioning/datasources
          name: grafana-datasources
        - mountPath: /etc/grafana
          name: grafana-config
      - env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io:443/acm-d/grafana-dashboard-loader-rhel9@sha256:7ef80b4d64f73cd51d0538bf68d0436ebb36280ce02ba86db9490f1f222593fd
        imagePullPolicy: IfNotPresent
        name: grafana-dashboard-loader
        resources:
          requests:
            cpu: 4m
            memory: 50Mi
        securityContext:
          privileged: false
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - args:
        - --provider=openshift
        - --upstream=http://localhost:3001
        - --https-address=:9443
        - --cookie-secret-file=/etc/proxy/secrets/session_secret
        - --cookie-expire=12h0m0s
        - --cookie-refresh=8h0m0s
        - '--openshift-delegate-urls={"/": {"resource": "projects", "verb": "list"}}'
        - --tls-cert=/etc/tls/private/tls.crt
        - --tls-key=/etc/tls/private/tls.key
        - --openshift-service-account=grafana
        - --pass-user-bearer-token=true
        - --pass-access-token=true
        - --client-id=grafana-proxy-client
        - --client-secret=grafana-proxy-client
        - --scope=user:full
        - --openshift-ca=/etc/pki/tls/cert.pem
        - --openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:069190846cfd4918ccbfbc0d06af2430c59935f9702c1047dfcb3726ad1d668c
        imagePullPolicy: IfNotPresent
        name: grafana-proxy
        ports:
        - containerPort: 9443
          name: public
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /oauth/healthz
            port: 9443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          privileged: false
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/tls/private
          name: tls-secret
        - mountPath: /etc/proxy/secrets
          name: cookie-secret
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: multiclusterhub-operator-pull-secret
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: grafana
      serviceAccountName: grafana
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: grafana-storage
      - name: grafana-datasources
        secret:
          defaultMode: 420
          secretName: grafana-datasources
      - name: grafana-config
        secret:
          defaultMode: 420
          secretName: grafana-config
      - name: tls-secret
        secret:
          defaultMode: 420
          secretName: grafana-tls
      - name: cookie-secret
        secret:
          defaultMode: 420
          secretName: rbac-proxy-cookie-secret
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: "2025-03-14T14:09:33Z"
    lastUpdateTime: "2025-03-14T14:09:33Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2025-03-14T14:08:53Z"
    lastUpdateTime: "2025-03-14T14:09:33Z"
    message: ReplicaSet "observability-grafana-797958984f" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

Version-Release number of selected component (if applicable):

ACM v2.13.0-RC4

How reproducible:

always for http_proxy cluster

Steps to Reproduce:

setup http_proxy cluster
install ACM and ACM observability
open grafana UI

Actual results:

500 error for grafana UI

Expected results:

no error

500 error for grafana UI in http_proxy cluster

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Details

Description

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Attachments

Easy Agile Planning Poker

Activity

People

Dates