Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-6782

Submariner - submariner-metrics-proxy pod crash on startup

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • None
    • ACM 2.9.0
    • None
    • False
    • None
    • False
    • No

      Description of problem:

      ACM 2.9/ Submariner 0.16.0

      The submariner-metrics-proxy pod crash on startup.

       

       

      Events:
        Type     Reason                 Age                   From               Message
        ----     ------                 ----                  ----               -------
        Normal   Scheduled              160m                  default-scheduler  Successfully assigned submariner-operator/submariner-metrics-proxy-5lggc to ip-10-0-64-117.us-east-2.compute.internal
        Warning  ErrorUpdatingResource  160m (x3 over 160m)   controlplane       addLogicalPort failed for submariner-operator/submariner-metrics-proxy-5lggc: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-0-64-117.us-east-2.compute.internal"
        Warning  NetworkNotReady        159m (x25 over 160m)  kubelet            network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
        Normal   AddedInterface         159m                  multus             Add eth0 [10.130.2.4/23] from ovn-kubernetes
        Warning  Failed                 75m                   kubelet            Error: container create failed: time="2023-08-07T10:01:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Warning  Failed                 40m                   kubelet            Error: container create failed: time="2023-08-07T10:36:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Normal   Pulled                 15m (x655 over 158m)  kubelet            Container image "registry.redhat.io/rhacm2/nettest-rhel8@sha256:0fab128d557b8803061846fdb70366a3b93d873648fc187eb9cc31471b67956e" already present on machine
        Warning  Failed                 10m                   kubelet            Error: container create failed: time="2023-08-07T11:06:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Warning  Failed                 5m8s                  kubelet            Error: container create failed: time="2023-08-07T11:11:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Warning  Failed                 8s                    kubelet            Error: container create failed: time="2023-08-07T11:16:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied" 

       

      Pod describe:

      Name:         submariner-metrics-proxy-5lggc
      Namespace:    submariner-operator
      Priority:     0
      Node:         ip-10-0-64-117.us-east-2.compute.internal/10.0.64.117
      Start Time:   Mon, 07 Aug 2023 11:36:59 +0300
      Labels:       app=submariner-metrics-proxy
                    component=metrics
                    controller-revision-hash=86447d97f6
                    pod-template-generation=1
      Annotations:  k8s.ovn.org/pod-networks:
                      {"default":{"ip_addresses":["10.130.2.4/23"],"mac_address":"0a:58:0a:82:02:04","gateway_ips":["10.130.2.1"],"ip_address":"10.130.2.4/23","...
                    k8s.v1.cni.cncf.io/network-status:
                      [{
                          "name": "ovn-kubernetes",
                          "interface": "eth0",
                          "ips": [
                              "10.130.2.4"
                          ],
                          "mac": "0a:58:0a:82:02:04",
                          "default": true,
                          "dns": {}
                      }]
                    openshift.io/scc: restricted-v2
                    seccomp.security.alpha.kubernetes.io/pod: runtime/default
      Status:       Pending
      IP:           10.130.2.4
      IPs:
        IP:           10.130.2.4
      Controlled By:  DaemonSet/submariner-metrics-proxy
      Containers:
        gateway-metrics-proxy:
          Container ID:  
          Image:         registry.redhat.io/rhacm2/nettest-rhel8@sha256:0fab128d557b8803061846fdb70366a3b93d873648fc187eb9cc31471b67956e
          Image ID:      
          Port:          <none>
          Host Port:     <none>
          Command:
            /app/metricsproxy
          Args:
            8080
            $(NODE_IP)
            32780
          State:          Waiting
            Reason:       CreateContainerError
          Ready:          False
          Restart Count:  0
          Environment:
            NODE_IP:   (v1:status.hostIP)
          Mounts:
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-96wmj (ro)
        globalnet-metrics-proxy:
          Container ID:  
          Image:         registry.redhat.io/rhacm2/nettest-rhel8@sha256:0fab128d557b8803061846fdb70366a3b93d873648fc187eb9cc31471b67956e
          Image ID:      
          Port:          <none>
          Host Port:     <none>
          Command:
            /app/metricsproxy
          Args:
            8081
            $(NODE_IP)
            32781
          State:          Waiting
            Reason:       CreateContainerError
          Ready:          False
          Restart Count:  0
          Environment:
            NODE_IP:   (v1:status.hostIP)
          Mounts:
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-96wmj (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        kube-api-access-96wmj:
          Type:                    Projected (a volume that contains injected data from multiple sources)
          TokenExpirationSeconds:  3607
          ConfigMapName:           kube-root-ca.crt
          ConfigMapOptional:       <nil>
          DownwardAPI:             true
          ConfigMapName:           openshift-service-ca.crt
          ConfigMapOptional:       <nil>
      QoS Class:                   BestEffort
      Node-Selectors:              submariner.io/gateway=true
      Tolerations:                 op=Exists
                                   node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                                   node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                                   node.kubernetes.io/not-ready:NoExecute op=Exists
                                   node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                                   node.kubernetes.io/unreachable:NoExecute op=Exists
                                   node.kubernetes.io/unschedulable:NoSchedule op=Exists
      Events:
        Type     Reason                 Age                   From               Message
        ----     ------                 ----                  ----               -------
        Normal   Scheduled              160m                  default-scheduler  Successfully assigned submariner-operator/submariner-metrics-proxy-5lggc to ip-10-0-64-117.us-east-2.compute.internal
        Warning  ErrorUpdatingResource  160m (x3 over 160m)   controlplane       addLogicalPort failed for submariner-operator/submariner-metrics-proxy-5lggc: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-0-64-117.us-east-2.compute.internal"
        Warning  NetworkNotReady        159m (x25 over 160m)  kubelet            network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
        Normal   AddedInterface         159m                  multus             Add eth0 [10.130.2.4/23] from ovn-kubernetes
        Warning  Failed                 75m                   kubelet            Error: container create failed: time="2023-08-07T10:01:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Warning  Failed                 40m                   kubelet            Error: container create failed: time="2023-08-07T10:36:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Normal   Pulled                 15m (x655 over 158m)  kubelet            Container image "registry.redhat.io/rhacm2/nettest-rhel8@sha256:0fab128d557b8803061846fdb70366a3b93d873648fc187eb9cc31471b67956e" already present on machine
        Warning  Failed                 10m                   kubelet            Error: container create failed: time="2023-08-07T11:06:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Warning  Failed                 5m8s                  kubelet            Error: container create failed: time="2023-08-07T11:11:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied"
        Warning  Failed                 8s                    kubelet            Error: container create failed: time="2023-08-07T11:16:59Z" level=error msg="runc create failed: unable to start container process: exec: \"/app/metricsproxy\": permission denied" 

       

            skitt@redhat.com Stephen Kitt
            nogatal Noga Tal (Inactive)
            Maxim Babushkin Maxim Babushkin
            ACM QE Team
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: