Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32304

[4.17] metal3-ironic-inspector CrashLoopBackOff - /certs/ca/ironic permission denied

XMLWordPrintable

    • Important
    • No
    • 2
    • Metal Platform 253, Metal Platform 254, Metal Platform 255, Metal Platform 256
    • 4
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required
    • Done
    • reproducible using "oc scale deployment -n openshift-machine-api metal3 --replicas=0”

      Description of problem:

      
      There is one pod of metal3 operator in constant failure state. The cluster was acting as Hub cluster with ACM + GitOps for SNO installation. It was working well for a few days, until this moment when no other sites could be deployed.
      
      oc get pods -A | grep metal3
      openshift-machine-api                              metal3-64cf86fb8b-fg5b9                                           3/4     CrashLoopBackOff   35 (108s ago)   155m
      openshift-machine-api                              metal3-baremetal-operator-84875f859d-6kj9s                        1/1     Running            0               155m
      openshift-machine-api                              metal3-image-customization-57f8d4fcd4-996hd                       1/1     Running            0               5h
      
          

      Version-Release number of selected component (if applicable):

      OCP version: 4.16.ec5
          

      How reproducible:

      Once it starts to fail, it does not recover.
          

      Steps to Reproduce:

          1. Unclear. Install Hub cluster with ACM+GitOps
          2. (Perhaps: Update AgentServiceConfig
          

      Actual results:

      Pod crashing and installation of spoke cluster fails
          

      Expected results:

      Pod running and installation of spoke cluster succeds.
          

      Additional info:

      Logs of metal3-ironic-inspector:
      
      `[kni@infra608-1 ~]$ oc logs pods/metal3-64cf86fb8b-fg5b9 -c metal3-ironic-inspector
      + CONFIG=/etc/ironic-inspector/ironic-inspector.conf
      + export IRONIC_INSPECTOR_ENABLE_DISCOVERY=false
      + IRONIC_INSPECTOR_ENABLE_DISCOVERY=false
      + export INSPECTOR_REVERSE_PROXY_SETUP=true
      + INSPECTOR_REVERSE_PROXY_SETUP=true
      + . /bin/tls-common.sh
      ++ export IRONIC_CERT_FILE=/certs/ironic/tls.crt
      ++ IRONIC_CERT_FILE=/certs/ironic/tls.crt
      ++ export IRONIC_KEY_FILE=/certs/ironic/tls.key
      ++ IRONIC_KEY_FILE=/certs/ironic/tls.key
      ++ export IRONIC_CACERT_FILE=/certs/ca/ironic/tls.crt
      ++ IRONIC_CACERT_FILE=/certs/ca/ironic/tls.crt
      ++ export IRONIC_INSECURE=true
      ++ IRONIC_INSECURE=true
      ++ export 'IRONIC_SSL_PROTOCOL=-ALL +TLSv1.2 +TLSv1.3'
      ++ IRONIC_SSL_PROTOCOL='-ALL +TLSv1.2 +TLSv1.3'
      ++ export 'IPXE_SSL_PROTOCOL=-ALL +TLSv1.2 +TLSv1.3'
      ++ IPXE_SSL_PROTOCOL='-ALL +TLSv1.2 +TLSv1.3'
      ++ export IRONIC_VMEDIA_SSL_PROTOCOL=ALL
      ++ IRONIC_VMEDIA_SSL_PROTOCOL=ALL
      ++ export IRONIC_INSPECTOR_CERT_FILE=/certs/ironic-inspector/tls.crt
      ++ IRONIC_INSPECTOR_CERT_FILE=/certs/ironic-inspector/tls.crt
      ++ export IRONIC_INSPECTOR_KEY_FILE=/certs/ironic-inspector/tls.key
      ++ IRONIC_INSPECTOR_KEY_FILE=/certs/ironic-inspector/tls.key
      ++ export IRONIC_INSPECTOR_CACERT_FILE=/certs/ca/ironic-inspector/tls.crt
      ++ IRONIC_INSPECTOR_CACERT_FILE=/certs/ca/ironic-inspector/tls.crt
      ++ export IRONIC_INSPECTOR_INSECURE=true
      ++ IRONIC_INSPECTOR_INSECURE=true
      ++ export IRONIC_VMEDIA_CERT_FILE=/certs/vmedia/tls.crt
      ++ IRONIC_VMEDIA_CERT_FILE=/certs/vmedia/tls.crt
      ++ export IRONIC_VMEDIA_KEY_FILE=/certs/vmedia/tls.key
      ++ IRONIC_VMEDIA_KEY_FILE=/certs/vmedia/tls.key
      ++ export IPXE_CERT_FILE=/certs/ipxe/tls.crt
      ++ IPXE_CERT_FILE=/certs/ipxe/tls.crt
      ++ export IPXE_KEY_FILE=/certs/ipxe/tls.key
      ++ IPXE_KEY_FILE=/certs/ipxe/tls.key
      ++ export RESTART_CONTAINER_CERTIFICATE_UPDATED=false
      ++ RESTART_CONTAINER_CERTIFICATE_UPDATED=false
      ++ export MARIADB_CACERT_FILE=/certs/ca/mariadb/tls.crt
      ++ MARIADB_CACERT_FILE=/certs/ca/mariadb/tls.crt
      ++ export IPXE_TLS_PORT=8084
      ++ IPXE_TLS_PORT=8084
      ++ mkdir -p /certs/ironic
      ++ mkdir -p /certs/ironic-inspector
      ++ mkdir -p /certs/ca/ironic
      mkdir: cannot create directory '/certs/ca/ironic': Permission denied
      
          

            rh-ee-masghar Mahnoor Asghar
            rlopezma@redhat.com Rodrigo Lopez Manrique
            Dmitry Dmitriev Dmitry Dmitriev
            Steeve Goveas
            Votes:
            0 Vote for this issue
            Watchers:
            23 Start watching this issue

              Created:
              Updated: