Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-33163

[IBI] The downloads pod in openshift-console NS is stuck in CreateContainerError state

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.16
    • LCA operator
    • None
    • No
    • False
    • Hide

      None

      Show
      None

      lifecycle-agent.v4.16.0
      OCP: 4.16.0-ec.6

      After successfully completing an IBI deployment, checking the pods status:

      oc get pod -A|grep -v Run|grep -v Comple
      NAMESPACE                                          NAME                                                         READY   STATUS                 RESTARTS      AGE
      openshift-console                                  downloads-79c8c6bd47-68lq9                                   0/1     CreateContainerError   0             16h
      
      
      oc describe pod -n openshift-console                                  downloads-79c8c6bd47-68lq9
      Name:                 downloads-79c8c6bd47-68lq9
      Namespace:            openshift-console
      Priority:             2000000000
      Priority Class Name:  system-cluster-critical
      Service Account:      default
      Node:                 api.target-0.qe.lab.redhat.com/192.168.125.130
      Start Time:           Tue, 30 Apr 2024 17:14:29 -0400
      Labels:               app=console
                            component=downloads
                            pod-template-hash=79c8c6bd47
      Annotations:          k8s.ovn.org/pod-networks:
                              {"default":{"ip_addresses":["10.129.0.23/23"],"mac_address":"0a:58:0a:81:00:17","gateway_ips":["10.129.0.1"],"routes":[{"dest":"10.128.0.0...
                            k8s.v1.cni.cncf.io/network-status:
                              [{
                                  "name": "ovn-kubernetes",
                                  "interface": "eth0",
                                  "ips": [
                                      "10.129.0.23"
                                  ],
                                  "mac": "0a:58:0a:81:00:17",
                                  "default": true,
                                  "dns": {}
                              }]
                            openshift.io/scc: restricted-v2
                            seccomp.security.alpha.kubernetes.io/pod: runtime/default
      Status:               Pending
      SeccompProfile:       RuntimeDefault
      IP:                   10.129.0.23
      IPs:
        IP:           10.129.0.23
      Controlled By:  ReplicaSet/downloads-79c8c6bd47
      Containers:
        download-server:
          Container ID:  
          Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0b3c4ef4f7503831a935466177f584295c1d58b75906dbb4eef2f54ca4c4d4b
          Image ID:      
          Port:          8080/TCP
          Host Port:     0/TCP
          Command:
            /bin/sh
          Args:
            -c
            cat <<EOF >>/tmp/serve.py
            import errno, http.server, os, re, signal, socket, sys, tarfile, tempfile, threading, time, zipfile
            
            signal.signal(signal.SIGTERM, lambda signum, frame: sys.exit(0))
            
            def write_index(path, message):
              with open(path, 'wb') as f:
                f.write('\n'.join([
                  '<!doctype html>',
                  '<html lang="en">',
                  '<head>',
                  '  <meta charset="utf-8">',
                  '</head>',
                  '<body>',
                  '  {}'.format(message),
                  '</body>',
                  '</html>',
                  '',
                ]).encode('utf-8'))
            
            # Launch multiple listeners as threads
            class Thread(threading.Thread):
              def __init__(self, i, socket):
                threading.Thread.__init__(self)
                self.i = i
                self.socket = socket
                self.daemon = True
                self.start()
            
              def run(self):
                server = http.server.SimpleHTTPRequestHandler
                server.server_version = "OpenShift Downloads Server"
                server.sys_version = ""
                httpd = http.server.HTTPServer(addr, server, False)
            
                # Prevent the HTTP server from re-binding every handler.
                # https://stackoverflow.com/questions/46210672/
                httpd.socket = self.socket
                httpd.server_bind = self.server_close = lambda self: None
            
                httpd.serve_forever()
            
            temp_dir = tempfile.mkdtemp()
            print('serving from {}'.format(temp_dir))
            os.chdir(temp_dir)
            for arch in ['amd64', 'arm64', 'ppc64le', 's390x']:
              os.mkdir(arch)
            content = ['<a href="oc-license">license</a>']
            os.symlink('/usr/share/openshift/LICENSE', 'oc-license')
            
            for arch, operating_system, path in [
                ('amd64', 'linux', '/usr/share/openshift/linux_amd64/oc'),
                ('amd64', 'mac', '/usr/share/openshift/mac/oc'),
                ('amd64', 'windows', '/usr/share/openshift/windows/oc.exe'),
                ('arm64', 'linux', '/usr/share/openshift/linux_arm64/oc'),
                ('arm64', 'mac', '/usr/share/openshift/mac_arm64/oc'),
                ('ppc64le', 'linux', '/usr/share/openshift/linux_ppc64le/oc'),
                ('s390x', 'linux', '/usr/share/openshift/linux_s390x/oc'),
                ]:
              basename = os.path.basename(path)
              target_path = os.path.join(arch, operating_system, basename)
              os.mkdir(os.path.join(arch, operating_system))
              os.symlink(path, target_path)
              base_root, _ = os.path.splitext(basename)
              archive_path_root = os.path.join(arch, operating_system, base_root)
              with tarfile.open('{}.tar'.format(archive_path_root), 'w') as tar:
                tar.add(path, basename)
              with zipfile.ZipFile('{}.zip'.format(archive_path_root), 'w') as zip:
                zip.write(path, basename)
              content.append(
                '<a href="{0}">oc ({1} {2})</a> (<a href="{3}.tar">tar</a> <a href="{3}.zip">zip</a>)'.format(
                  target_path, arch, operating_system, archive_path_root
                )
              )
            
            for root, directories, filenames in os.walk(temp_dir):
              root_link = os.path.relpath(temp_dir, os.path.join(root, 'child')).replace(os.path.sep, '/')
              for directory in directories:
                write_index(
                  path=os.path.join(root, directory, 'index.html'),
                  message='<p>Directory listings are disabled.  See <a href="{}">here</a> for available content.</p>'.format(root_link),
                )
            
            write_index(
              path=os.path.join(temp_dir, 'index.html'),
              message='\n'.join(
                ['<ul>'] +
                ['  <li>{}</li>'.format(entry) for entry in content] +
                ['</ul>']
              ),
            )
            
            # Create socket
            # IPv6 should handle IPv4 passively so long as it is not bound to a
            # specific address or set to IPv6_ONLY
            # https://stackoverflow.com/questions/25817848/python-3-does-http-server-support-ipv6
            try:
              addr = ('::', 8080)
              sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
            except socket.error as err:
              # errno.EAFNOSUPPORT is "socket.error: [Errno 97] Address family not supported by protocol"
              # When IPv6 is disabled, socket will bind using IPv4.
              if err.errno == errno.EAFNOSUPPORT:
                addr = ('', 8080)
                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
              else:
                raise    
            sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            sock.bind(addr)
            sock.listen(5)
            
            [Thread(i, socket=sock) for i in range(100)]
            time.sleep(9e9)
            EOF
            exec python3 /tmp/serve.py
            
          State:          Waiting
            Reason:       CreateContainerError
          Ready:          False
          Restart Count:  0
          Requests:
            cpu:        10m
            memory:     50Mi
          Liveness:     http-get http://:8080/ delay=0s timeout=1s period=10s #success=1 #failure=3
          Readiness:    http-get http://:8080/ delay=0s timeout=1s period=10s #success=1 #failure=3
          Environment:  <none>
          Mounts:
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m5dt2 (ro)
      Conditions:
        Type                        Status
        PodReadyToStartContainers   True 
        Initialized                 True 
        Ready                       False 
        ContainersReady             False 
        PodScheduled                True 
      Volumes:
        kube-api-access-m5dt2:
          Type:                    Projected (a volume that contains injected data from multiple sources)
          TokenExpirationSeconds:  3607
          ConfigMapName:           kube-root-ca.crt
          ConfigMapOptional:       <nil>
          DownwardAPI:             true
          ConfigMapName:           openshift-service-ca.crt
          ConfigMapOptional:       <nil>
      QoS Class:                   Burstable
      Node-Selectors:              kubernetes.io/os=linux
      Tolerations:                 node-role.kubernetes.io/master:NoSchedule op=Exists
                                   node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                                   node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                   node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
      Events:
        Type    Reason  Age                   From     Message
        ----    ------  ----                  ----     -------
        Normal  Pulled  26s (x4532 over 16h)  kubelet  Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0b3c4ef4f7503831a935466177f584295c1d58b75906dbb4eef2f54ca4c4d4b" already present on machine
      
      

            itsoiref@redhat.com Igal Tsoiref
            achuzhoy@redhat.com Alexander Chuzhoy
            Alexander Chuzhoy Alexander Chuzhoy
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: