Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1392

Quay build worker pod turns to error on openshift

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • quay-v3.4.0
    • quay-v3.4.0
    • quay-builder
    • None
    • 0

    Description

      Description of problem:

      When launch build worker on openshift, the worker pod turns to error

      Version-Release number of selected component (if applicable):

      registry-proxy.engineering.redhat.com/rh-osbs/quay-quay@sha256:c0d9ebd1e83b300579bf01dc3d6cf4104adb1f8c93b1e7c5ddbc0a09df3c8eb1
        registry-proxy.engineering.redhat.com/rh-osbs/quay-quay:v3.4.0-24

      How reproducible:

      Always

      Steps to Reproduce:
      1.Deploy Quay by operator

      2. Create project 'builder' in OCP
      3.Create build trigger
      4.Check build status

      Actual results:

      build is stuck due to no build worker launched

      Expected results:

      build is completed with ephemeral build worker

      Additional info:

      $ oc get po -w -n builder
      NAME                                               READY   STATUS   RESTARTS   AGE
      367b3dfe-2657-4311-837f-698e634ec7dc-hvwj7-45psc   0/1     Error    0          75s
      
      

       

      build worker pod logs:

      $ oc logs -f 367b3dfe-2657-4311-837f-698e634ec7dc-plljm-h52p4 -n builder
      + set -o nounset
      + mkdir -p /userdata/openstack/latest
      + echo '{"ignition": {"version": "3.0.0"}, "passwd": {"users": [{"groups": ["sudo", "docker"], "name": "core"}]}, "storage": {"files": [{"path": "/etc/pki/ca-trust/source/anchors/ssl.cert", "contents": {"source": "data:,-----BEGIN%20CERTIFICATE-----%0AMIIEMDCCAxigAwIBAgIBAjANBgkqhkiG9w0BAQsFADBYMVYwVAYDVQQDDE10ZXN0%0ALXF1YXkxLXF1YXktcXVheS5hcHBzLnByZXNlcnZlLWR5YW4uMTIxNS01ejUucWUu%0AcmhjbG91ZC5jb20tY2FAMTYwODE5MDY2MjAeFw0yMDEyMTcwNjM3NDJaFw0yMTEy%0AMTcwNjM3NDJaMFUxUzBRBgNVBAMMSnRlc3QtcXVheTEtcXVheS1xdWF5LmFwcHMu%0AcHJlc2VydmUtZHlhbi4xMjE1LTV6NS5xZS5yaGNsb3VkLmNvbUAxNjA4MTkwNjYz%0AMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmAPXf7H4YdU6XNKX%2B1Ti%0AXralurqbQf0C1LI7SYWFIud%2BuTZGCDyjx7Oph/tMRy0zO7q9MxoT0ISW8hCj/AgJ%0AhRosYuNQdnRnuZuwN8cUApr67ghudtK6iMmypR%2BHDLkZFlppULG9p/j4kmVXL7GU%0ADLx13vJt8fe2hHungHh9p27G8/cuo408f1XFOX4ky08aB2WOaRx22a6ozIWAWtQh%0A33soDIIwGJd0yrADf5UHEWFqrZfZzSPE46OVJadi2wIRcMihIT/hb6nv%2BdQuspJ8%0AQP%2BI098BrZY%2B/gwnYQoVCqaBMuaW3R4ouBCJr%2B65i13GStujThKngQ5GUEJRcaNz%0AgwIDAQABo4IBBjCCAQIwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF%0ABwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAU6/6TKPoYMvE%2Bcue2Ws%2BJlJJu%0AJvIwgasGA1UdEQSBozCBoII/dGVzdC1xdWF5MS1xdWF5LXF1YXkuYXBwcy5wcmVz%0AZXJ2ZS1keWFuLjEyMTUtNXo1LnFlLnJoY2xvdWQuY29tghN0ZXN0LXF1YXkxLXF1%0AYXktYXBwghx0ZXN0LXF1YXkxLXF1YXktYXBwLnF1YXkuc3Zjgip0ZXN0LXF1YXkx%0ALXF1YXktYXBwLnF1YXkuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQELBQAD%0AggEBAJq4y48XS31q8gxcObFhBTXvjB%2BVRKlOokcwFeUNplg7s/UzPOYJVGNzjOtp%0As7uoXmNMZZBxRq6T60a8LK7WizTcWEkS/6ry7ju8kmtmNp9gCZyRgRm1cvEbekjv%0AxZAWxPmVlPZc4upWUlvgMq2KXlkdMZj7d8WvoxIVvq50vocVPSS4HxHTkCjACTgY%0AO00WtgAov/l9tNtp9060R3IZpvWGjkDpOqaukYrXC2NNjqMnWseCLjMFaXpZRIPL%0AxP2tQMKjBa0E82T%2B7M09CqvbAhA2vEbpjpd6nEFzqZS0mc%2Bz9Psgi1lT%2BeiZKjME%0APr7Q2Z0%2BROYvh82RtGcdRx2XxFQ%3D%0A-----END%20CERTIFICATE-----%0A-----BEGIN%20CERTIFICATE-----%0AMIIDbTCCAlWgAwIBAgIBATANBgkqhkiG9w0BAQsFADBYMVYwVAYDVQQDDE10ZXN0%0ALXF1YXkxLXF1YXktcXVheS5hcHBzLnByZXNlcnZlLWR5YW4uMTIxNS01ejUucWUu%0AcmhjbG91ZC5jb20tY2FAMTYwODE5MDY2MjAeFw0yMDEyMTcwNjM3NDJaFw0yMTEy%0AMTcwNjM3NDJaMFgxVjBUBgNVBAMMTXRlc3QtcXVheTEtcXVheS1xdWF5LmFwcHMu%0AcHJlc2VydmUtZHlhbi4xMjE1LTV6NS5xZS5yaGNsb3VkLmNvbS1jYUAxNjA4MTkw%0ANjYyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAztOD9mS0R3DH1cTg%0AgzN3b3y5Rx67pFx3maUJ0p38/PYlMZkf6nPe4YqpseYX3FI7R6zCYUU7dFaBYssz%0AYBaUl94YuBNZJQcnIksVzBrXfQ1k4p9s8A3C5my25tYsYcHqtbcvA9kzIY/TG5Lx%0AO36W2oPdigq9L9rd3fneC%2B5lQH7NZ1%2BYvzQOhzJhmilSIdJReys%2BfzMvv7Ki%2BxFj%0Ah0lH15b1zOG7MuRlsuqLY3Rox7oAs0xBFFY8sz8/rHv%2B8Up1dvwWTKmMsA6Eh1K2%0AQJqbOWlT0n7f2aL%2BpsbSQm2VLy2PJplXg4Tui1oaH64Yk9W8oxLqScDS74te8vfC%0A/PnhOQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAd%0ABgNVHQ4EFgQU6/6TKPoYMvE%2Bcue2Ws%2BJlJJuJvIwDQYJKoZIhvcNAQELBQADggEB%0AAEGBe0G6Wgsdz6AEp9MyM%2BDwjuqSwxel9a4m4GTbXr4OMQypRCLVoI0aiCQBbLam%0AWx4qQt145DEhGv9qpPHG5pUsdAy7I/mqdha6F9RLjTYoNhy5gD9PtaFKyVd6SG9T%0ALaVgmsOmymu3lnLa9zAYC8x69OSfLeRZwwZ4ix1RnHSHH/H4TSwK7OJcd9/igk4m%0AdaAu0Hok/kPF7ssTV6yg0NsUOPp7hcW%2BYrFefkFxOIr6WwBRxwoqtr0Hds9zGb0%2B%0AevVCGfjNi%2BgRku26CK%2BEmrhNd0QasaR6CpGKZrGd/w%2BiWgGtj1%2BvaKuJ/lR1VMce%0AbcBz5mqhemSSEuAcIrzUqVM%3D%0A-----END%20CERTIFICATE-----%0A"}, "mode": 420}, {"path": "/etc/hostname", "contents": {"source": "data:,367b3dfe-2657-4311-837f-698e634ec7dc"}, "mode": 420}, {"overwrite": true, "path": "/etc/zincati/config.d/90-disable-auto-updates.toml", "contents": {"source": "data:,%5Bupdates%5D%0Aenabled%20%3D%20false"}, "mode": 420}, {"path": "/usr/local/bin/disable-aws-metadata.sh", "contents": {"source": "data:,%23%21/bin/bash%0Aiptables%20-t%20nat%20-I%20PREROUTING%20-p%20tcp%20-d%20169.254.169.254%20--dport%2080%20-j%20DNAT%20--to-destination%201.1.1.1"}, "mode": 493}, {"path": "/root/overrides.list", "contents": {"source": "data:,TOKEN%3DeyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Ijk1MTc3NTRhNmY3MDk0OTI1M2UxYmU3OGRiNDdhZTIzMjEwMGQ3ZTg3MjJlMTFhOTFlNzUzMDA2YzlhOTFiODcifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoidGVzdC1xdWF5MS1xdWF5LXF1YXkuYXBwcy5wcmVzZXJ2ZS1keWFuLjEyMTUtNXo1LnFlLnJoY2xvdWQuY29tIiwibmJmIjoxNjA4MTk0MDgzLCJpYXQiOjE2MDgxOTQwODMsImV4cCI6MTYwODE5NDU4Mywic3ViIjoiKGFub255bW91cykiLCJhY2Nlc3MiOnt9LCJjb250ZXh0Ijp7InRva2VuX3R5cGUiOiJidWlsZF9qb2JfcmVnaXN0cmF0aW9uIiwiYnVpbGRfaWQiOiIzNjdiM2RmZS0yNjU3LTQzMTEtODM3Zi02OThlNjM0ZWM3ZGMiLCJqb2JfaWQiOiJidWlsZG1hbi9wcm9kdWN0aW9uL2J1aWxkaW5nLzM2N2IzZGZlLTI2NTctNDMxMS04MzdmLTY5OGU2MzRlYzdkYyIsImV4cGlyYXRpb24iOjUwMH19.zgEYQhPqkzWeVU4PqYEqhIuO-yCYqSVA5NNMTHXYJ4JpX12Dc4Xd9c9jhH9M8hdiriVJ4IyAXYRJVvaLa6Ijhf-p7xn9HRw38ugDHpAQxKyntq4oqVqjFRXpQZHmgHq5GlJO_06ywizpt6j9Z87RUT1Q5W5CdNwkvE2Xg3CL2wr4iAvfOYfsf3lXqCNf1lzykgdLWq8qrAOpJ9_0dD2jAORankhKHZGtVXRpe_B837I8pkHLbJhl-t69JwKmd7Qot9emFB6sgdAKyNlXTUCIMl30U_Qd9zUnt8hWlHgsp5_1M9evN_RrH1LJrtu3vjWKmpGqALAg_O9bv4iWzkYK2g%0ASERVER%3Dtest-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com%3A50051"}, "mode": 420}]}, "systemd": {"units": [{"name": "quay-builder.service", "enabled": true, "contents": "[Unit]\n\nAfter=docker.service\nRequires=docker.service\n\n\n\n\n\n\n[Service]\n\nRestart=no\nTimeoutStartSec=600\nTimeoutStopSec=2000\n\n\n\nExecStartPre=/usr/bin/docker login -u root -p quayroot brew.registry.redhat.io\n\nExecStart=/usr/bin/docker run --rm --net=host --privileged --env-file /root/overrides.list -v /var/run/docker.sock:/var/run/docker.sock -v /etc/pki/ca-trust-source/anchors:/certs --name quay-builder brew.registry.redhat.io/rh-osbs/quay-quay-builder:v3.4.0-9 \n\nExecStop=/usr/bin/docker stop quay-builder\n\nExecStopPost=/bin/sh -xc \"/bin/sleep 120; /usr/bin/systemctl --no-block poweroff\"\n\n\n[Install]\nWantedBy=multi-user.target"}, {"name": "systemd-journal-gatewayd.socket", "enabled": true, "contents": "[Unit]\nDescription=Journal Gateway Service Socket\n[Socket]\nListenStream=/var/run/journald.sock\nService=systemd-journal-gatewayd.service\n[Install]\nWantedBy=sockets.target"}, {"name": "disable-aws-metadata.service", "enabled": true, "contents": "[Unit]\nDescription=Disable AWS metadata service\nBefore=network-pre.target\nWants=network-pre.target\n[Service]\nType=oneshot\nExecStart=/usr/local/bin/disable-aws-metadata.sh\nRemainAfterExit=yes\n[Install]\nWantedBy=multi-user.target"}, {"name": "machine-lifetime.service", "enabled": true, "contents": "[Unit]\nDescription=Machine Lifetime Service\n[Service]\nType=oneshot\nExecStart=/bin/sh -xc \"/bin/sleep 10800; /usr/bin/systemctl --no-block poweroff\"\n[Install]\nWantedBy=multi-user.target"}]}}'
      + qemu-img resize ./coreos_production_qemu_image.qcow2 8G
      qemu-img: Unable to initialize gcryptreal	0m0.009s
      user	0m0.005s
      sys	0m0.004s

      Attachments

        1. builder0118.log
          104 kB
        2. image-2020-12-18-17-02-33-319.png
          image-2020-12-18-17-02-33-319.png
          361 kB
        3. image-2020-12-24-11-16-52-640.png
          image-2020-12-24-11-16-52-640.png
          319 kB
        4. image-2021-01-05-11-18-31-994.png
          image-2021-01-05-11-18-31-994.png
          328 kB
        5. image-2021-01-06-17-08-51-141.png
          image-2021-01-06-17-08-51-141.png
          335 kB
        6. log
          129 kB

        Issue Links

          Activity

            People

              sleesinc Kenny Lee Sin Cheong
              rhn-support-dyan Dongbo Yan
              Dongbo Yan Dongbo Yan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: