Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-4474

[Upstream] Failed to create Tempo Monolithic instance when tls is enabled

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • rhosdt-3.3
    • None
    • Tempo
    • None
    • Tracing Sprint # 257

      Version of components:
      OCP version: 4.16.0-0.nightly-2024-07-17-183709
      tempo-operator.v0.11.1

      Description of the problem:
      When a Tempo Monolithic instance is created with TLS enabled and without setting any custom certs, the instance fails to create with error ''

      MountVolume.SetUp failed for volume "tempo-mono-cert-serving-cert" : secret "tempo-mono-cert-serving-cert" not found
      
      
      % oc get pods
      NAME                     READY   STATUS              RESTARTS   AGE
      minio-744f658d8d-c5rxq   1/1     Running             0          27s
      tempo-mono-cert-0        0/2     ContainerCreating   0          7s
       
      % oc describe pod tempo-mono-cert-0
      Events:
        Type     Reason       Age               From               Message
        ----     ------       ----              ----               -------
        Normal   Scheduled    17s               default-scheduler  Successfully assigned chainsaw-tls-mono-st/tempo-mono-cert-0 to ip-10-0-13-81.us-east-2.compute.internal
        Warning  FailedMount  2s (x6 over 17s)  kubelet            MountVolume.SetUp failed for volume "tempo-mono-cert-serving-cert" : secret "tempo-mono-cert-serving-cert" not found
      
      % oc get cm
      NAME                               DATA   AGE
      kube-root-ca.crt                   1      15m
      openshift-service-ca.crt           1      15m
      tempo-mono-cert-config             2      15m
      tempo-mono-cert-serving-cabundle   1      15m
      
      % oc get secrets 
      NAME                              TYPE                      DATA   AGE
      builder-dockercfg-vlwb4           kubernetes.io/dockercfg   1      15m
      default-dockercfg-tdw4f           kubernetes.io/dockercfg   1      15m
      deployer-dockercfg-vrgkv          kubernetes.io/dockercfg   1      15m
      minio                             Opaque                    4      15m
      tempo-mono-cert-dockercfg-pbr9f   kubernetes.io/dockercfg   1      15m 

      Steps to reproduce the issue:
      1. Install the Tempo operator built off the latest upstream branch.
      2. Run the following steps to create the monolithic instance.

      oc new-project chainsaw-tls-mono-st
      
      cat 00-install-storage.yaml        
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        labels:
          app.kubernetes.io/name: minio
        name: minio
        namespace: chainsaw-tls-mono-st
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 2Gi
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: minio
        namespace: chainsaw-tls-mono-st
      spec:
        selector:
          matchLabels:
            app.kubernetes.io/name: minio
        strategy:
          type: Recreate
        template:
          metadata:
            labels:
              app.kubernetes.io/name: minio
          spec:
            containers:
              - command:
                  - /bin/sh
                  - -c
                  - |
                    mkdir -p /storage/tempo && \
                    minio server /storage
                env:
                  - name: MINIO_ACCESS_KEY
                    value: tempo
                  - name: MINIO_SECRET_KEY
                    value: supersecret
                image: minio/minio
                name: minio
                ports:
                  - containerPort: 9000
                volumeMounts:
                  - mountPath: /storage
                    name: storage
            volumes:
              - name: storage
                persistentVolumeClaim:
                  claimName: minio
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: minio
        namespace: chainsaw-tls-mono-st
      spec:
        ports:
          - port: 9000
            protocol: TCP
            targetPort: 9000
        selector:
          app.kubernetes.io/name: minio
        type: ClusterIP
      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: minio
        namespace: chainsaw-tls-mono-st
      stringData:
        endpoint: http://minio:9000
        bucket: tempo
        access_key_id: tempo
        access_key_secret: supersecret
      type: Opaque
      
      oc create -f 00-install-storage.yaml
      
      cat 01-install-tempo.yaml            
      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoMonolithic
      metadata:
        name: mono-cert
        namespace: chainsaw-tls-mono-st
      spec:
        jaegerui:
          enabled: true
        ingestion:
          otlp:
            grpc:
              tls:
                enabled: true
            http:
              tls:
                enabled: true
      
      oc create -f 01-install-tempo.yaml

      3. Check the monolithic pod status. Its stuck in ContainerCreating state with the mount error in the pod events. 

       

              rvargasp@redhat.com Ruben Vargas Palma
              rhn-support-ikanse Ishwar Kanse
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: