Uploaded image for project: 'Cluster Observability Operator'
  1. Cluster Observability Operator
  2. COO-1263

ObservabilityInstaller fails to read TLS CA ConfigMap: "ConfigMap not found" despite resource existing

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • cluster-observability
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description:

      Summary

      ObservabilityInstaller with TLS configuration fails to create TempoStack due to operator inability to read existing CA ConfigMap, resulting in continuous reconciliation errors.

      Bug Details

      Location: `pkg/controllers/observability/tempo_components.go:99-105`

      The `tempoStackSecrets` function fails to read the CA ConfigMap specified in `spec.capabilities.tracing.storage.objectStorage.tls.caConfigMap` with error: failed to get object storage CA configmap custom-ca: ConfigMap "custom-ca" not found

      Steps to Reproduce

        ### 1. Create MinIO storage backend with TLS certificates

        Apply the following resources to set up the S3-compatible storage backend:

        apiVersion: v1
  kind: Secret
  metadata:
    name: minio-cert
    namespace: observability-operator
  data:
    private.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUpRZ0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQ1N3d2dna29BZ0VBQW9JQ0FRQ2
  RUSkFjazV6OTZadDcKbEFmb2FramdBU1lvTktrSlRwSm1SVEFCUDkwWHdMOE1GMmt1MHRkVXZXV0xyMlpiKzhBZVB4YjNrS1l6dS9BNQozUDlENVdxNjhubmV
  wOHJUMUMzTXlObzNrVVQzbXFJdTVpSGl3NTRVUmpUTjFRNjltQTBVZ084OGlMOElmdFFUCndidk92UHY2dlFsMlNlK1k0R2hiMzVGN0kvOGZaYkI5cDRINUk0
  THRQTXlkblM4K1dPeWZRdHJDVXIxcVhLakIKOHE2SXZLTjRrUUhLZyswems0d1lHN3g0S2wyV2dXd2d3dDRiTjRvUU5XMlhmY3NNZzBqZ0hNUGJLY3hxQWtHZ
  ApsbEE2QUJFTmp1SW9qc2hubkhhYy81MWFENzlnN1BFVkdnWnRtaUorZjBXSllHbFZvOVFuTXUyVHptVmtuU0IrCi8remlJbmRwdDFiYUhIQWhmaXhmWHJtRF
  pvSVlWbVlRVG1uaDB0a05SZVF2MGNJWTU4YjVjWWYzUWJVK0Z3aHoKN0Q0OVlRNjhoelFSU3FMTk1DdWhpa3I5WXdVMWoyQ1VJaVcrdVgzUFRxRTVlenBnYTk
  1YUVLdVlud0dKdHp1UgpKUk4yR0tKRStlSVFWY1l0c1hhNmNHQTlmZ0s1WElodldYM1ZLVkZETTZLRVRMRjIvT0FTMDlDSXBDQ3Q5QXcvCkpZbGptSzZ4bXR4
  YU1yeGdKNHZldWJseC90Q2dnMDhuUjJTNkZhZjlPOHl3bTdSOWk4ZGNjL29GcjRFclpJQUQKU3FwTXJPMmh5V0J2d1dRakhqeDVsY3UxR09oM3RWRCtnb05NN
  jczMDU5WThwWVF3VThJd296aE5VUUxPQTlwcApDNmZhdm9QL2ZWQlVwUUQxeW13ekxFSUJDUkoxZHdJREFRQUJBb0lDQUVBMmJKQVYzRjAwKzZGZmhUam1Cbm
  pYClR6M1RmWHZNcE92MTY0cGZDVUQwa0tqOWQ1VkRRZDhZYi9jdGM2SXlLN1BOT25GMmpicUp5cCs4YlVKSXlhQ20KY29Tbng1OWd1dytob0o5VmdLdDVaeUt
  4Rkgwdk1QMkN5WFNMT0xqQjlQajlSMnhWUUhMVjMzcHZPRjFGZmJIVwpjS0htdmRZUktkanpjNzZxcTYra0FmL1JuVGZZdk01ZTV3S2lrNUltbC9YTncyTFho
  dlovKys0c3hQNFJ6OFBjCm9qZ0xVZzRmZWUydmlPa2ZGbFJsNGYrZnZTNnNKZ0xHNU1XV2R3alZJUzlIM3RZOWduSjJHRllkOGdNeldzTEgKcE9BOHBXZlNrZ
  00yUUFSdmNFYy9nbzBFTWtLVXpPY0p5cGV4a1Y2Q0FnMjhkd3JERTFZVzVKaWpuZzZVVmwxZwpJemhwR1Q2dVR4UzQyTjFXRW9xazFyS1k0amNNVENTTmgvZ0
  hkZmltZFZTOXJ1RjdJRlkrMGkybHFQd0xrNjlOClBZNE1CaWE0b0VvdUREMUxObXArTWI0R0k3dlVadzlQK0hYSFBiS2RyM2xzOVh5Y28wOURHOFdQT0NqV2d
  qVU0KL0E2azBPUk4zYmdYT0t4M1pwSm5nckhRWnJjUTZ4dFFlWWRZb2lwUE5PNmhGWHZzQkhBdm8ydnYzSXpJWGpnbgpmZFR5SWpiOENPNUUydlJyb0oxU3JK
  dXYvQ1hsTHZRNGxzU0xXdmVtZUg2aFIrQkZ0SXduK2Y1K2J1aUVvQ2dJCk9FQm0yb29rZGpFNWc0Vml4ODhpbEJmM0ltb1ZKY2JNRmN4M05Ob1NYWHNpQTNqa
  VFkanpLUHFocExuUkFlMnEKaUw1a1FXdU5ENk9oVG1pUjJaU3BBb0lCQVFEUWVCeit4Tm1IUDQzRG5rVVQzRkxZMHJOek8yNXhqRFZoMXRsawpCblBYUGJ6cH
  plVVZnc2pGYmlZcEFrUy9CQkJXZjdZcTNnWnJsS0MvOUJqVlNZem5memFZK3FEOHRPOWFjN0NHCk9YTWFPUFFYMWNXWmxkcG9sRDlEWjRlNTlqVGNHV1hWSmF
  tQUgyOEJxdm41T25jTGFNKytmRWlBaVowUVlIeHoKelgrMTc2QUdJeDhWTjRpT0RFM1RuVnJTL201Mk9aVS9wOEJ2WXB0cWp1Y3pBSk5XaTJVeTgwOUljTEts
  cnJxeQpTVU9HcXBQd1BWVTEzMDdSMk5odWJXZC9vK3FDVjFWejRzZEhnRVk0cXhSakpCYzZyV3NvZW5rcVlqNDNIQzVGCkp0MXBiaXJuYXAxWlZJSFVzTUZ0S
  U5UQkFLakpnMXFRaW5QVlFuNTZlOEhGQzhUSkFvSUJBUURCS2NORk9HczUKcDluZWlFMTA0Wms3SXNwcks1NTFlV2duWDV3a292ZmNSdjFNbEthRXNRV1R2Uk
  FaQ0RKeFNvQ3pXV3BzaGZIdwpuSGdBSlZLcTVGOU16QXgzQmYrNU5QeC9WSVFkQUdsanY5MWJ1YXdPUDZlY0lkWnl4cldBSmZXNVFLaDNLdm55Ci8vRHRGejM
  2UktRNWQrbDhVRStXOU0xWEttVzNUSWptMkwwSjBRU1h0TEZLRk9jSkc2Z3gxeUE3NUp6Mk1GdXAKMCs5RjMyam5DNWdVQS9uWFpUa016OXVkdXFuWDFnUE5z
  cWk0TUVrRlZGcjJjZXdBMXZISWxYT3BFei9VcTZobwpubGJiZ0RQTGFpd1dBRk54cW5kOHRuWFVTeEpMZVFTck9kb0V0bFhwdG9GME1xTWFBVkJHbDd6ZnZ4Z
  mJzVnBrCkJDQ09FQ2gxOE1nL0FvSUJBQUdhYXQ2cXRPR05ZNUcxamVKeE4xSGpqTzI5NzBPQVNJalFGaXZtd00xTzlEejQKRUdMeDlvbE9KNHo1TDM0b2tkZV
  ljOVl3anV3VTRJZVpZUGVDOFVwMmQvMEZNR2ZCSEhZZG5hWHFtekFBY0NBZQp6TDQ1b014ZEYzOTFORjRCdnJaU202M2Y5RVcvK0NLVEpnajhvaDI4V0p5bWF
  BaFZCcGd4cGNOZU9UVTArbzRSCmh4T3FzYW9sL2hNNmlwQkUxS1NrSExtNG1JcEN2OTg1NmgxQ1M5L0dncWk1cGtwd1UyOWl3SnpPWTI2TnNVV1AKc3pBTzM1
  Uk5qcHlLeXRqNGNiMCszckZkZFJyZnFhcVY1am84OU5ER3RLUkRETnQrbGV2bWs1U0VqLzlYa1l6bQpEUCs2M3FTOEYwQzNsK3RFRGdnK3NsdWFOUG5XNFJJZ
  0VZVnFhY0VDZ2dFQkFKL2F5L3NSQWN1cEdTaU1iRzE1CjliRjJ6VVY3eDVzNGFmV0xybEN6ZmFRU0s3V2duMTNZa2VEOTZjQ05oQkUxc3VTNTd0VDQxSXhuTV
  BnTjZ5ZjUKN3RSRFdWMTIzWGFNWHhtUSsvUjBUc042R05xOEFhT1pFa0NuMEMwVVBVaGNJY3E1czdGTEk2YnlJSlJWVTdrMQo3U0tTRmV3SEdwYTI2T2hsOGQ
  rTHV4QXY5ZUdWMU5NbmVrRHdvM0pxTFB0eUtQN241dTNtZnpWQWF4Zk1GNmg5CmM5RzhxSnFVN20ybjV2dlZhcEZ3VDUxL2FSbGsvRWtvMGs1NVlQSWc3OThy
  cjVnUWtoYVNLTURMcU4zbWJ2YTgKV3RlcVZXUVB5dXF1cFR5WEZBRFptZitKMU45cGRoZ0FMWW5ZRGhxTFU3L1prenEvamlNRjF1UkdkaHFiVEtDdQpMMGtDZ
  2dFQVQxR0FpWnQzWHN5OGFxWm9xY1lCL1I0L1M2YVMzL1psczlGSVFoTko1SVhnWFhLR2FNSThSbFQ3CkM2bmZVeVFYUnZVVFNveHpSelhhNXEvWWxMRS9pZm
  paeXcvUFdCMENDQ2xlWGhSaHU1aWJWcXc0ZHdndDZPcnIKdkJrdnEvMStSck5NbVFacFhLMkJrZjBQMFhwd3ZCVVphNlpaL2hjM1BpY091NUNLdURXWGMzZ2h
  jTmdyeGJBcwo3WldWMEpvUkJYTjJKS3piQk5TYUEyUGNPaFpFQ0Z0c21VeWQxZzhWRnl5U1VPbzFGdU9wNmxGTGttSTEyRUduCkdwcldPdkFKREhwNnNzVEY0
  Y3ZlVGlSRFFyWkpBaFlXTTlmdmR5TVJZVzlJUUlFVFZ1SzJxZjV0d3Q3RTR2ZlQKdmtTZ29McFNTTWxMT29DeGRyNHIxMUxTU1h1bGpBPT0KLS0tLS1FTkQgU
  FJJVkFURSBLRVktLS0tLQo=
    public.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZGakNDQXY2Z0F3SUJBZ0lVVWtzZEFKU20rYzZCWGtTNU43NDFYZzlvS3g4d0RRWUp
  Lb1pJaHZjTkFRRUwKQlFBd0V6RVJNQThHQTFVRUF3d0lUWGxFWlcxdlEwRXdIaGNOTWpNd09ESTBNVFl4TVRBM1doY05Nek13T0RJeApNVFl4TVRBM1dqQVFN
  UTR3REFZRFZRUUREQVZ0YVc1cGJ6Q0NBaUl3RFFZSktvWklodmNOQVFFQkJRQURnZ0lQCkFEQ0NBZ29DZ2dJQkFKMU1rQnlUblAzcG0zdVVCK2hxU09BQkppZ
  zBxUWxPa21aRk1BRS8zUmZBdnd3WGFTN1MKMTFTOVpZdXZabHY3d0I0L0Z2ZVFwak83OERuYy8wUGxhcnJ5ZWQ2bnl0UFVMY3pJMmplUlJQZWFvaTdtSWVMRA
  puaFJHTk0zVkRyMllEUlNBN3p5SXZ3aCsxQlBCdTg2OCsvcTlDWFpKNzVqZ2FGdmZrWHNqL3g5bHNIMm5nZmtqCmd1MDh6SjJkTHo1WTdKOUMyc0pTdldwY3F
  NSHlyb2k4bzNpUkFjcUQ3VE9UakJnYnZIZ3FYWmFCYkNEQzNoczMKaWhBMWJaZDl5d3lEU09BY3c5c3B6R29DUVoyV1VEb0FFUTJPNGlpT3lHZWNkcHovblZv
  UHYyRHM4UlVhQm0yYQpJbjUvUllsZ2FWV2oxQ2N5N1pQT1pXU2RJSDcvN09JaWQybTNWdG9jY0NGK0xGOWV1WU5tZ2hoV1poQk9hZUhTCjJRMUY1Qy9Sd2hoV
  Hh2bHhoL2RCdFQ0WENIUHNQajFoRHJ5SE5CRktvczB3SzZHS1N2MWpCVFdQWUpRaUpiNjUKZmM5T29UbDdPbUJyM2xvUXE1aWZBWW0zTzVFbEUzWVlva1Q1NG
  hCVnhpMnhkcnB3WUQxK0FybGNpRzlaZmRVcApVVU16b29STXNYYjg0QkxUMElpa0lLMzBERDhsaVdPWXJyR2EzRm95dkdBbmk5NjV1WEgrMEtDRFR5ZEhaTG9
  WCnAvMDd6TENidEgyTHgxeHorZ1d2Z1N0a2dBTktxa3lzN2FISllHL0JaQ01lUEhtVnk3VVk2SGUxVVA2Q2cwenIKdmZUbjFqeWxoREJUd2pDak9FMVJBczRE
  Mm1rTHA5cStnLzk5VUZTbEFQWEtiRE1zUWdFSkVuVjNBZ01CQUFHagpaVEJqTUIwR0ExVWREZ1FXQkJTRlhKTmtrZm5TKzQzNmFNNDRqc0hoSkxlMkF6QWZCZ
  05WSFNNRUdEQVdnQlJxCldqUXE4QXgyT0Nkd1AwVmRWdzJaMHNsclZUQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CQUdBMVVkRVFRSk1BZUMKQlcxcGJtbHZNQT
  BHQ1NxR1NJYjNEUUVCQ3dVQUE0SUNBUUJnOXprWHFPdSt5Vm02Uk9GQWVaTmZwN0FSYlZ1TQpFbm52WW5JYWRDQ3dVZktialZweWlzL0ZrVEhBV3NHT3F5end
  Zc1lVdXdkS050clplbjNCYk1HbVA0OUJzWnRUCnFyZXQ5NGF2RXo1aCs4L1RWY3ZRNG5uN2pUak9jSDFuUmFyUStTdk9aTGUzSnY3emM2ZVorS3JoR1FWQlB4
  TGsKQll5bHlzN3AxM3RkWjZPK3VBZkxFdVZxY0oranIzWGdpY1RUdlA0U2pZdVNPclFWdGNKMDBHZ2xPRzl5ZktuMQpmWlV3RjVvVG1meU9JWXNLS1pGTnEvN
  2Yra1o0Q0dOVmwraXNPOGIxTkEyTS9VZEVoZjVoMExOaFM2YTNOb2VECjdJaFNZZnhETDZOUzVpM2N0MGR2MTRtd1l4Nys4c0tNeUNvcnJiUGF0ZEZJdlVia0
  9KS3dpR1RyNElVZHF0eW0Kc25WenVId0xORW1RZGFtY3R4Tis4Ykc1RWFyNEdpMUw5SXpDRTdDSXI5WExIMFhkNVo4YUJpSGFTNVB0Ry9iVwpKWTlWSlJQSXl
  aZi9idkNPbVFhZUZIQ3ltcXBJOU5xWnBUU3dDT3hVdVZpeHJ6ZE9oamlIREI5RVA0MDcxeTFZCmFUbDJBS2VxdHFmOFlmd1RSR04zZkdrbXlEb1UvTWtERFhq
  bWFVZGEzNmVJVE9NTFJ6REtaWEM2UWpQTDl4YlUKcU5DMWR2Mjl0ZUM2V1pjWTE3L3VIaWhlVVBwdENJNEIvY3lVOUpyVG13VjhCbFRlNCs1UmZGTjZCZUQ4R
  Gl3eQpTNjRIRUZIaW9PUVBjUlJQb1NqNVhXMnNsTFpiMkczV1RwcWVyZDdrdmlBRFhOMXovZ0lIbUZlaDdpdE5jR1hnCjNLWWUwendaOEJSNG1RPT0KLS0tLS
  1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: minio
    namespace: observability-operator
  spec:
    selector:
      matchLabels:
        app.kubernetes.io/name: minio
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app.kubernetes.io/name: minio
      spec:
        containers:
          - command:
              - /bin/sh
              - -c
              - |
                mkdir -p /storage/tempo && \
                minio server --certs-dir=/root/.minio/certs /storage
            env:
              - name: MINIO_ACCESS_KEY
                value: tempo
              - name: MINIO_SECRET_KEY
                value: supersecret
            image: quay.io/minio/minio:latest
            name: minio
            ports:
              - containerPort: 9000
            volumeMounts:
              - mountPath: /storage
                name: storage
              - name: cert
                mountPath: /root/.minio/certs
        volumes:
          - name: storage
            emptyDir: {}
          - name: cert
            secret:
              secretName: minio-cert
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: minio
    namespace: observability-operator
  spec:
    ports:
      - port: 9000
        protocol: TCP
        targetPort: 9000
    selector:
      app.kubernetes.io/name: minio
    type: ClusterIP
  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: minio
    namespace: observability-operator
  stringData:
    endpoint: https://minio:9000
    bucket: tempo
    access_key_id: tempo
    access_key_secret: supersecret
  type: Opaque

        2. Create CA ConfigMap for TLS verification

        apiVersion: v1
  kind: ConfigMap
  metadata:
    name: custom-ca
    namespace: observability-operator
  data:
    ca.crt: |
      -----BEGIN CERTIFICATE-----
      MIIFBzCCAu+gAwIBAgIUDwTaPC/j59gRvcgVyvWn0Qd8WaUwDQYJKoZIhvcNAQEL
      BQAwEzERMA8GA1UEAwwITXlEZW1vQ0EwHhcNMjMwODI0MTYxMTAyWhcNMzMwODIx
      MTYxMTAyWjATMREwDwYDVQQDDAhNeURlbW9DQTCCAiIwDQYJKoZIhvcNAQEBBQAD
      ggIPADCCAgoCggIBAInKlIOllf/hGmh8v87UZyygzDrnvmP1wgRxYYcKPDNzQo4T
      CwSeSl8kCmaSmTZ7ii98BUiLoMS1VkHVkVa3xyK/hmSAjwL4sbWBlkEtuA1b0s8v
      Hl6nA5EZpGkDKWeJI5vGCLa+YNyuN2iXe7fxz5MXQZ8Rbr1ADGLHp5CG+kyXjuLt
      nmUO8JwT9NnaOzkmI5sRpePJLGqLLboxzqxiYrNtME1NB1ogjyro7nWCgqIRqAlQ
      EzJ+PyjQ7ILEgQ3DGYOfxvvwf80WSudsTjU0Ek4QGgSPjylQ/lLXLNTh7xCVvSRS
      b5FR9UQ1Gy8QH3MZyv0YOphoKN7VNcgdU/9ykSeC7us8q54wGR3/HQlPOcs/6uwQ
      /5VFajPZrkrX7Lx/8XsIcyAYdsM+WcRhErIL/FZjYard26x7t4fDbUZ2Z/+nvD43
      jAMoxi4uTyAwMbMxw+Co83zBrsFZEYWN8p8mJ1zpZDudhoAccB8wfHw0t5pvomOV
      GQ9wIyjJdQcu26wo1DXCNhxrS9lrWaZvSqAYvvsnlvbnocUvoQVlxYyptA879VEJ
      Uvs+t292w6Qmf5VnT/o9BsdkRht6eX3Wb095dpNuczTRLgyoS7gcxenj2YOs0dML
      8POhzv/xuuPvj+Y5ZaKTdBe92sRgxvewH4YdxfAzElepmyJjo4wPYnIVLFAFAgMB
      AAGjUzBRMB0GA1UdDgQWBBRqWjQq8Ax2OCdwP0VdVw2Z0slrVTAfBgNVHSMEGDAW
      gBRqWjQq8Ax2OCdwP0VdVw2Z0slrVTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
      DQEBCwUAA4ICAQBZnpF4wjLYJmqTjST8Eqz5YNjTms9J7M6ClmG8CABmgmnjfmHb
      anwfk+sJxOkobo5i7lWtbqvaNUyNIcLnLSEYP3BDibBQl2qJ7tTPz3ZlXUw8dSsf
      LqOr5BbA4k468tcCEo8sIeZPEFiZdQe+ovPVz/626x/GjOd0XYyGsFC40qkZJ51c
      NPrI5XLEmshH0s6EKzER+O/1iZzRwuCencTRa7xfDZV4ueKA9lLGenrsPYUuUsoq
      8082hO9IaNtScRdrDyw9fZKK2OTWtrsydcA/0Vf0afzrcYJwLv0PMsDK32FXUjHH
      bMAj9Rn1yAq17CZlK75rhDoI8EHHPmZ3CxdwqCfTxUzEA/pBt0aGs0Lpq1BC1gUt
      rdde09M5BwzzbN4wAOCSTmr2EC9Xq3JNOvyusIRnTrv+yP5JBfZCDd7j3+AQ+KSW
      3zgss/f2FSxm83E9vZYRX2ETT6xCkcVpyDpl/MT+5qrP2RRpNuu7sN/qvMI6fLFX
      CdtUMc8S3tQAzOiLXAcA0TmT8NlW3bxpf2tpOCfIG9t9EhNjSq/Iw+uAt581ITta
      xtdKFBv1LLup2pr/hbmbP+0XHNFuUK36I7oantHhagXL/pGO3vcugppAd+YNzOW9
      Qr67VgKeixOrNiK6W9FzuaWYadv0XOgrJbFhsvmtYqpEMDGZzfdb5dAzdA==
      -----END CERTIFICATE-----

        3. Create ObservabilityInstaller with TLS configuration. 
      Note if you have installed the observability operator using operator-sdk, you need to the set the operator subscription install mode to Automatic in the subscription so that the OTEL and Tempo operators can be installed. Otherwise we need to manually approve the installplan for the operators. 

        apiVersion: observability.openshift.io/v1alpha1
  kind: ObservabilityInstaller
  metadata:
    name: simplest
    namespace: observability-operator
  spec:
    capabilities:
      tracing:
        enabled: true
        operators:
          install: true
        storage:
          objectStorage:
            s3:
              accessKeyID: "tempo"
              accessKeySecret:
                name: "minio"
                key: "access_key_secret"
              bucket: "tempo"
              endpoint: "https://minio:9000"
              region: "us-east-2"
            tls:
              caConfigMap:
                name: "custom-ca"
                key: "ca.crt"
              minVersion: "1.2"

        4. Observe the error

        Check operator logs: 

       kubectl logs -n observability-operator -l name=observability-operator

        Expected error output:

        failed to create TempoStack secret: failed to get object storage CA configmap custom-ca: ConfigMap "custom-ca" not found

       Expected Behavior

        Operator should successfully read the CA ConfigMap and create the TempoStack with TLS configuration enabled.

       Actual Behavior

        Operator fails with "ConfigMap not found" error in continuous reconciliation loop, preventing TempoStack creation.

      Verification

        - ✅ ConfigMap exists: kubectl get configmap custom-ca -n observability-operator

        - ✅ RBAC permissions: Operator has get/list/watch permissions on configmaps

        - ✅ Namespace correct: Both resources in same namespace

        - ✅ CR syntax: TLS configuration follows API schema

        - ❌ Operator can read ConfigMap through controller-runtime client

        Root Cause Analysis

        This appears to be a controller-runtime client caching issue where the operator's cached client cannot properly read the

        ConfigMap despite:

        - Proper RBAC permissions

        - Correct namespace and resource naming

        - ConfigMap existing before and after operator restarts

        Failed Workarounds Attempted

        - Restarting operator pod

        - Recreating ConfigMap

        - Recreating ObservabilityInstaller CR

        - Different resource creation ordering

        Suggested Fix

        1. Use direct API client instead of cached client for ConfigMap lookup in tempoStackSecrets

        2. Add retry logic with exponential backoff for resource reads

        3. Implement proper cache warming for dependent resources

        4. Add debug logging to troubleshoot client cache state

        Code Reference

        File: pkg/controllers/observability/tempo_components.go

        Function: tempoStackSecrets

        Lines: 99-105

      The Jira issue description is assisted by Claude code. 

              ploffay@redhat.com Pavol Loffay
              rhn-support-ikanse Ishwar Kanse
              None
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: