Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-5716

tempo-gateway-opo container failing due to missing default requests and limits

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • rhosdt-3.6, rhosdt-3.7
    • Tempo
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • Important

      Tested this on 4.19.10 cluster

      configured resource quota in tempo-test namespace

      Deployed tempostack.

      The Gateway deployment dont show any resources configured

      the deployment defines two containers:

      tempo-gateway → has resources defined (requests and limits)

      tempo-gateway-opa → currently does NOT have resources set (empty resources: {})

      The resource quota requires all containers to have CPU and memory requests and limits set.
      kind: Deployment
      apiVersion: apps/v1
      metadata:
      annotations:
      deployment.kubernetes.io/revision: '2'
      resourceVersion: '464429'
      name: tempo-instance-gateway
      uid: c0b267ad-0a5d-4052-9214-ec442c918bdb
      creationTimestamp: '2025-10-21T14:56:57Z'
      generation: 2
      managedFields:

      • manager: kube-controller-manager
        operation: Update
        apiVersion: apps/v1
        time: '2025-10-21T14:56:57Z'
        fieldsType: FieldsV1
        fieldsV1:
        'f:metadata':
        'f:annotations':
        .: {}
        'f:deployment.kubernetes.io/revision': {}
        'f:status':
        'f:conditions':
        .: {}
        'k: {"type":"Available"}

        ':
        .: {}
        'f:lastTransitionTime': {}
        'f:lastUpdateTime': {}
        'f:message': {}
        'f:reason': {}
        'f:status': {}
        'f:type': {}
        'k:

        {"type":"Progressing"}

        ':
        .: {}
        'f:lastTransitionTime': {}
        'f:lastUpdateTime': {}
        'f:message': {}
        'f:reason': {}
        'f:status': {}
        'f:type': {}
        'f:observedGeneration': {}
        'f:replicas': {}
        'f:unavailableReplicas': {}
        'f:updatedReplicas': {}
        subresource: status

      • manager: tempo-operator
        operation: Update
        apiVersion: apps/v1
        time: '2025-10-21T14:56:57Z'
        fieldsType: FieldsV1
        fieldsV1:
        'f:metadata':
        'f:labels':
        .: {}
        'f:app.kubernetes.io/component': {}
        'f:app.kubernetes.io/instance': {}
        'f:app.kubernetes.io/managed-by': {}
        'f:app.kubernetes.io/name': {}
        'f:ownerReferences':
        .: {}
        'k: {"uid":"693b63b6-8267-4f34-8bd6-3a735d1b8960"}

        ': {}
        'f:spec':
        'f:progressDeadlineSeconds': {}
        'f:replicas': {}
        'f:revisionHistoryLimit': {}
        'f:selector': {}
        'f:strategy':
        'f:rollingUpdate':
        .: {}
        'f:maxSurge': {}
        'f:maxUnavailable': {}
        'f:type': {}
        'f:template':
        'f:metadata':
        'f:annotations':
        .: {}
        'f:tempo.grafana.com/config.hash': {}
        'f:tempo.grafana.com/rbacConfig.hash': {}
        'f:tempo.grafana.com/tenantsConfig.hash': {}
        'f:labels':
        .: {}
        'f:app.kubernetes.io/component': {}
        'f:app.kubernetes.io/instance': {}
        'f:app.kubernetes.io/managed-by': {}
        'f:app.kubernetes.io/name': {}
        'f:spec':
        'f:volumes':
        .: {}
        'k:

        {"name":"cabundle"}

        ':
        .: {}
        'f:configMap':
        .: {}
        'f:defaultMode': {}
        'f:name': {}
        'f:name': {}
        'k:

        {"name":"rbac"}

        ':
        .: {}
        'f:configMap':
        .: {}
        'f:defaultMode': {}
        'f:items': {}
        'f:name': {}
        'f:name': {}
        'k:

        {"name":"serving-certs"}

        ':
        .: {}
        'f:name': {}
        'f:secret':
        .: {}
        'f:defaultMode': {}
        'f:secretName': {}
        'k:

        {"name":"tempo-instance-ca-bundle"}

        ':
        .: {}
        'f:configMap':
        .: {}
        'f:defaultMode': {}
        'f:name': {}
        'f:name': {}
        'k:

        {"name":"tempo-instance-gateway-mtls"}

        ':
        .: {}
        'f:name': {}
        'f:secret':
        .: {}
        'f:defaultMode': {}
        'f:secretName': {}
        'k:

        {"name":"tenant"}

        ':
        .: {}
        'f:name': {}
        'f:secret':
        .: {}
        'f:defaultMode': {}
        'f:items': {}
        'f:secretName': {}
        'f:containers':
        'k:

        {"name":"tempo-gateway"}

        ':
        'f:image': {}
        'f:volumeMounts':
        .: {}
        'k:

        {"mountPath":"/etc/tempo-gateway/cabundle"}

        ':
        .: {}
        'f:mountPath': {}
        'f:name': {}
        'f:readOnly': {}
        'k:

        {"mountPath":"/etc/tempo-gateway/cm"}

        ':
        .: {}
        'f:mountPath': {}
        'f:name': {}
        'f:readOnly': {}
        'k:

        {"mountPath":"/etc/tempo-gateway/secret/tenants.yaml"}

        ':
        .: {}
        'f:mountPath': {}
        'f:name': {}
        'f:readOnly': {}
        'f:subPath': {}
        'k:

        {"mountPath":"/etc/tempo-gateway/serving-certs"}

        ':
        .: {}
        'f:mountPath': {}
        'f:name': {}
        'f:readOnly': {}
        'k:

        {"mountPath":"/var/run/ca"}

        ':
        .: {}
        'f:mountPath': {}
        'f:name': {}
        'k:

        {"mountPath":"/var/run/tls/server"}

        ':
        .: {}
        'f:mountPath': {}
        'f:name': {}
        'f:terminationMessagePolicy': {}
        .: {}
        'f:resources':
        .: {}
        'f:limits':
        .: {}
        'f:cpu': {}
        'f:memory': {}
        'f:requests':
        .: {}
        'f:cpu': {}
        'f:memory': {}
        'f:args': {}
        'f:livenessProbe':
        .: {}
        'f:failureThreshold': {}
        'f:httpGet':
        .: {}
        'f:path': {}
        'f:port': {}
        'f:scheme': {}
        'f:periodSeconds': {}
        'f:successThreshold': {}
        'f:timeoutSeconds': {}
        'f:env':
        .: {}
        'k:

        {"name":"GOMEMLIMIT"}

        ':
        .: {}
        'f:name': {}
        'f:value': {}
        'f:readinessProbe':
        .: {}
        'f:failureThreshold': {}
        'f:httpGet':
        .: {}
        'f:path': {}
        'f:port': {}
        'f:scheme': {}
        'f:periodSeconds': {}
        'f:successThreshold': {}
        'f:timeoutSeconds': {}
        'f:securityContext':
        .: {}
        'f:allowPrivilegeEscalation': {}
        'f:capabilities':
        .: {}
        'f:drop': {}
        'f:readOnlyRootFilesystem': {}
        'f:terminationMessagePath': {}
        'f:imagePullPolicy': {}
        'f:ports':
        .: {}
        'k:

        {"containerPort":8080,"protocol":"TCP"}

        ':
        .: {}
        'f:containerPort': {}
        'f:name': {}
        'f:protocol': {}
        'k:

        {"containerPort":8081,"protocol":"TCP"}

        ':
        .: {}
        'f:containerPort': {}
        'f:name': {}
        'f:protocol': {}
        'k:

        {"containerPort":8090,"protocol":"TCP"}

        ':
        .: {}
        'f:containerPort': {}
        'f:name': {}
        'f:protocol': {}
        'f:name': {}
        'k:

        {"name":"tempo-gateway-opa"}

        ':
        'f:image': {}
        'f:terminationMessagePolicy': {}
        .: {}
        'f:resources': {}
        'f:args': {}
        'f:livenessProbe':
        .: {}
        'f:failureThreshold': {}
        'f:httpGet':
        .: {}
        'f:path': {}
        'f:port': {}
        'f:scheme': {}
        'f:periodSeconds': {}
        'f:successThreshold': {}
        'f:timeoutSeconds': {}
        'f:readinessProbe':
        .: {}
        'f:failureThreshold': {}
        'f:httpGet':
        .: {}
        'f:path': {}
        'f:port': {}
        'f:scheme': {}
        'f:periodSeconds': {}
        'f:successThreshold': {}
        'f:timeoutSeconds': {}
        'f:terminationMessagePath': {}
        'f:imagePullPolicy': {}
        'f:ports':
        .: {}
        'k:

        {"containerPort":8082,"protocol":"TCP"}

        ':
        .: {}
        'f:containerPort': {}
        'f:name': {}
        'f:protocol': {}
        'k:

        {"containerPort":8083,"protocol":"TCP"}

        ':
        .: {}
        'f:containerPort': {}
        'f:name': {}
        'f:protocol': {}
        'f:name': {}
        'f:dnsPolicy': {}
        'f:serviceAccount': {}
        'f:restartPolicy': {}
        'f:schedulerName': {}
        'f:terminationGracePeriodSeconds': {}
        'f:serviceAccountName': {}
        'f:securityContext': {}
        namespace: tempo-test
        ownerReferences:

      • apiVersion: tempo.grafana.com/v1alpha1
        kind: TempoStack
        name: instance
        uid: 693b63b6-8267-4f34-8bd6-3a735d1b8960
        controller: true
        blockOwnerDeletion: true
        labels:
        app.kubernetes.io/component: gateway
        app.kubernetes.io/instance: instance
        app.kubernetes.io/managed-by: tempo-operator
        app.kubernetes.io/name: tempo
        spec:
        replicas: 1
        selector:
        matchLabels:
        app.kubernetes.io/component: gateway
        app.kubernetes.io/instance: instance
        app.kubernetes.io/managed-by: tempo-operator
        app.kubernetes.io/name: tempo
        template:
        metadata:
        creationTimestamp: null
        labels:
        app.kubernetes.io/component: gateway
        app.kubernetes.io/instance: instance
        app.kubernetes.io/managed-by: tempo-operator
        app.kubernetes.io/name: tempo
        annotations:
        tempo.grafana.com/config.hash: 250382f2bd20945074c42b0dc88842a75c64113bb49b5f45dd907ecbe30a2e0b
        tempo.grafana.com/rbacConfig.hash: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        tempo.grafana.com/tenantsConfig.hash: 968825c1d5b13bfd2ca9c94eb9cafe453ae59fefb5dc59b4a53b3f812bbdf6b2
        spec:
        restartPolicy: Always
        serviceAccountName: tempo-instance-gateway
        schedulerName: default-scheduler
        terminationGracePeriodSeconds: 30
        securityContext: {}
        containers:
      • resources:
        limits:
        cpu: 120m
        memory: '107374184'
        requests:
        cpu: 36m
        memory: '32212256'
        readinessProbe:
        httpGet:
        path: /ready
        port: internal
        scheme: HTTPS
        timeoutSeconds: 1
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 12
        terminationMessagePath: /dev/termination-log
        name: tempo-gateway
        livenessProbe:
        httpGet:
        path: /live
        port: internal
        scheme: HTTPS
        timeoutSeconds: 2
        periodSeconds: 30
        successThreshold: 1
        failureThreshold: 10
        env:
      • name: GOMEMLIMIT
        value: '85899347'
        securityContext:
        capabilities:
        drop:
      • ALL
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        ports:
      • name: grpc-public
        containerPort: 8090
        protocol: TCP
      • name: internal
        containerPort: 8081
        protocol: TCP
      • name: public
        containerPort: 8080
        protocol: TCP
        imagePullPolicy: IfNotPresent
        volumeMounts:
      • name: rbac
        readOnly: true
        mountPath: /etc/tempo-gateway/cm
      • name: tenant
        readOnly: true
        mountPath: /etc/tempo-gateway/secret/tenants.yaml
        subPath: tenants.yaml
      • name: tempo-instance-ca-bundle
        mountPath: /var/run/ca
      • name: tempo-instance-gateway-mtls
        mountPath: /var/run/tls/server
      • name: serving-certs
        readOnly: true
        mountPath: /etc/tempo-gateway/serving-certs
      • name: cabundle
        readOnly: true
        mountPath: /etc/tempo-gateway/cabundle
        terminationMessagePolicy: File
        image: 'registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:533f1f5698cd6e7660cb0ee8d4eceaf076e88cf9f61854ae6dae7136e9e60d77'
        args:
      • '--traces.tenant-header=x-scope-orgid'
      • '--web.listen=0.0.0.0:8080'
      • '--web.internal.listen=0.0.0.0:8081'
      • '--traces.write.otlpgrpc.endpoint=tempo-instance-distributor.tempo-test.svc.cluster.local:4317'
      • '--traces.write.otlphttp.endpoint=https://tempo-instance-distributor.tempo-test.svc.cluster.local:4318'
      • '--traces.write-timeout=30s'
      • '--traces.tempo.endpoint=https://tempo-instance-query-frontend.tempo-test.svc.cluster.local:3200'
      • '--grpc.listen=0.0.0.0:8090'
      • '--rbac.config=/etc/tempo-gateway/cm/rbac.yaml'
      • '--tenants.config=/etc/tempo-gateway/secret/tenants.yaml'
      • '--log.level=info'
      • '--tls.internal.server.key-file=/var/run/tls/server/tls.key'
      • '--tls.internal.server.cert-file=/var/run/tls/server/tls.crt'
      • '--traces.tls.key-file=/var/run/tls/server/tls.key'
      • '--traces.tls.cert-file=/var/run/tls/server/tls.crt'
      • '--traces.tls.ca-file=/var/run/ca/service-ca.crt'
      • '--traces.tls.watch-certs=true'
      • '--tls.server.cert-file=/etc/tempo-gateway/serving-certs/tls.crt'
      • '--tls.server.key-file=/etc/tempo-gateway/serving-certs/tls.key'
      • '--tls.healthchecks.server-ca-file=/etc/tempo-gateway/cabundle/service-ca.crt'
      • '--tls.healthchecks.server-name=tempo-instance-gateway.tempo-test.svc.cluster.local'
      • '--web.healthchecks.url=https://localhost:8080'
      • '--tls.client-auth-type=NoClientCert'
      • '--traces.read.endpoint=https://tempo-instance-query-frontend.tempo-test.svc.cluster.local:16686'
      • resources: {}. <----------------
        readinessProbe:
        httpGet:
        path: /ready
        port: 8083
        scheme: HTTP
        timeoutSeconds: 1
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 12
        terminationMessagePath: /dev/termination-log
        name: tempo-gateway-opa
        livenessProbe:
        httpGet:
        path: /live
        port: 8083
        scheme: HTTP
        timeoutSeconds: 2
        periodSeconds: 30
        successThreshold: 1
        failureThreshold: 10
        ports:
      • name: public
        containerPort: 8082
        protocol: TCP
      • name: opa-metrics
        containerPort: 8083
        protocol: TCP
        imagePullPolicy: IfNotPresent
        terminationMessagePolicy: File
        image: 'registry.redhat.io/rhosdt/tempo-gateway-opa-rhel8@sha256:cd28b6205b8c9861721c7bc7aa806319f50a15c86b7b197858e502946565818c'
        args:
      • '--log.level=warn'
      • '--web.listen=:8082'
      • '--web.internal.listen=:8083'
      • '--web.healthchecks.url=http://localhost:8082'
      • '--opa.package=tempostack'
      • '--openshift.mappings=default=tempo.grafana.com'
        serviceAccount: tempo-instance-gateway
        volumes:
      • name: rbac
        configMap:
        name: tempo-instance-gateway
        items:
      • key: rbac.yaml
        path: rbac.yaml
        defaultMode: 420
      • name: tenant
        secret:
        secretName: tempo-instance-gateway
        items:
      • key: tenants.yaml
        path: tenants.yaml
        defaultMode: 420
      • name: tempo-instance-ca-bundle
        configMap:
        name: tempo-instance-ca-bundle
        defaultMode: 420
      • name: tempo-instance-gateway-mtls
        secret:
        secretName: tempo-instance-gateway-mtls
        defaultMode: 420
      • name: serving-certs
        secret:
        secretName: tempo-instance-gateway-tls
        defaultMode: 420
      • name: cabundle
        configMap:
        name: tempo-instance-gateway-cabundle
        defaultMode: 420
        dnsPolicy: ClusterFirst
        strategy:
        type: RollingUpdate
        rollingUpdate:
        maxUnavailable: 25%
        maxSurge: 25%
        revisionHistoryLimit: 10
        progressDeadlineSeconds: 600
        status:
        observedGeneration: 2
        replicas: 2
        updatedReplicas: 1
        unavailableReplicas: 2
        conditions:
      • type: Available
        status: 'False'
        lastUpdateTime: '2025-10-21T14:56:57Z'
        lastTransitionTime: '2025-10-21T14:56:57Z'
        reason: MinimumReplicasUnavailable
        message: Deployment does not have minimum availability.
      • type: Progressing
        status: 'True'
        lastUpdateTime: '2025-10-21T14:56:57Z'
        lastTransitionTime: '2025-10-21T14:56:57Z'
        reason: ReplicaSetUpdated
        message: ReplicaSet "tempo-instance-gateway-54f5598659" is progressing.
        I see below events

      51s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-tmlmb" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      51s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-9jzsr" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      51s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-zmjzz" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      51s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-7448n" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      51s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-sfsbv" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      51s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-b4p2h" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      50s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-znpgg" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      50s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-hljtt" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      49s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd Error creating: pods "tempo-instance-gateway-7669f6f4cd-lcxzf" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      10s Warning FailedCreate replicaset/tempo-instance-gateway-7669f6f4cd (combined from similar events): Error creating: pods "tempo-instance-gateway-7669f6f4cd-95t2v" is forbidden: failed quota: tempo-test-quota: must specify limits.cpu for: tempo-gateway-opa; limits.memory for: tempo-gateway-opa; requests.cpu for: tempo-gateway-opa; requests.memory for: tempo-gateway-opa
      9m57s Normal ScalingReplicaSet deployment/tempo-instance-gateway Scaled up replica set tempo-instance-gateway-746cbcf4 from 0 to 1
      9m57s Normal ScalingReplicaSet deployment/tempo-instance-gateway Scaled up replica set tempo-instance-gateway-54f5598659 from 0 to 1
      8m6s Normal ScalingReplicaSet deployment/tempo-instance-gateway Scaled down replica set tempo-instance-gateway-746cbcf4 from 1 to 0
      51s Normal ScalingReplicaSet deployment/tempo-instance-gateway Scaled up replica set tempo-instance-gateway-7669f6f4cd from 0 to 1

      Tested this on latest version as well 0.18.0

              Unassigned Unassigned
              rhn-support-samrinha SAYED AMRIN HANIF
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: