Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-254

[2112353] lifecycle.posStart hook does not have network connectivity.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 4.10
    • None
    • Moderate
    • None
    • False
    • Hide

      None

      Show
      None

      +++ This bug was initially created as a clone of Bug #2081562 +++

      Description of problem:

      lifecycle.posStart does not have network connectivity on OpenShiftSDN CNI. (OVNKubernetes does not have the issue)

      Version-Release number of selected component (if applicable):
      4.10

      How reproducible:
      always

      Steps to Reproduce:
      1. create statefulset
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      $ oc create -f statefulset.yaml
      $ cat statefulset.yaml
      apiVersion: apps/v1
      kind: StatefulSet
      metadata:
      name: httpd
      spec:
      serviceName: "httpd"
      replicas: 1
      selector:
      matchLabels:
      app: httpd
      template:
      metadata:
      labels:
      app: httpd
      spec:
      containers:

      • name: httpd
        image: registry.redhat.io/rhel8/httpd-24:1-191
        ports:
      • containerPort: 80
        name: web
        lifecycle:
        postStart:
        exec:
        command:
      • /bin/sh
      • -c
      • curl -k https://<IP:PORT> > /tmp/urltest.txt
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      Actual results:

      PostStartHook fails
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      36s Normal Killing pod/httpd-0 FailedPostStartHook
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      Expected results:

      PostStartHook should not fail.

      Additional info:

      by adding a dummy initContainers, you can workaround the issue.
      something like this:
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      spec:
      initContainers:

      • name: init-myservice
        image: busybox:1.28
        command: ['sh', '-c', 'sleep 2']
        containers:
      • name: httpd
        image: registry.redhat.io/rhel8/httpd-24:1-191
        ports:
      • containerPort: 80
        name: web
        lifecycle:
        postStart:
        exec:
        command:
      • /bin/sh
      • -c
      • curl -k <IP:PORT> > /tmp/urltest.txt
        ....
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      — Additional comment from rphillips@redhat.com on 2022-05-11 19:48:10 UTC —

      crio's contract with networking is to have networking up when the container starts. Moving to the openshift-sdn team to help triage what is going on.

      — Additional comment from hyoskim@redhat.com on 2022-06-09 00:40:33 UTC —

      Hello,

      Is there any update on this issue?

      — Additional comment from npinaeva@redhat.com on 2022-06-09 07:53:38 UTC —

      Hello, yeah we found the root cause and working on the fix now - PR should be ready by the end of the week

      — Additional comment from aos-team-art-private@bot.bugzilla.redhat.com on 2022-07-24 15:21:48 UTC —

      Elliott changed bug status from MODIFIED to ON_QA.
      This bug is expected to ship in the next 4.11 release.

      — Additional comment from errata-xmlrpc@redhat.com on 2022-07-27 00:18:40 UTC —

      This bug has been added to advisory RHSA-2022:5069 by OpenShift Release Team Bot (ocp-build/buildvm.openshift.eng.bos.redhat.com@REDHAT.COM)

      — Additional comment from swasthan@redhat.com on 2022-07-27 05:38:55 UTC —

      Hello Team, thank you for the help so far!

      May we know if this is going to backport in v4.10.z as well?

      Regards,
      Swadeep

      — Additional comment from zzhao@redhat.com on 2022-07-27 06:40:30 UTC —

      this fixed PR is merged to build 4.12.0-0.nightly-2022-07-24-180529
      So I update the target version to 4.12 version.

      — Additional comment from zzhao@redhat.com on 2022-07-27 06:48:33 UTC —

      still failed on build 4.12.0-0.nightly-2022-07-26-131732

      Creating above statefulset and pod still cannot be worked with same error

      27s Warning FailedPostStartHook pod/httpd-0 Exec lifecycle hook ([/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt]) for Container "httpd" in Pod "httpd-0_default(7e519841-7092-4513-928b-03c7783ddc7d)" failed - error: command '/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt' exited with 1: /bin/sh: -c: line 0: syntax error near unexpected token `>'...
      85s Normal Killing pod/httpd-0 FailedPostStartHook

      — Additional comment from npinaeva@redhat.com on 2022-07-27 12:50:53 UTC —

      Hey @zzhao@redhat.com can you share full statefulset yaml you're running?
      Doesn't "line 0: syntax error near unexpected token `>'..." mean bash command is wrong?

      — Additional comment from zzhao@redhat.com on 2022-07-27 13:36:43 UTC —

      (In reply to Nadia Pinaeva from comment #9)
      > Hey @zzhao@redhat.com can you share full statefulset yaml you're running?
      > Doesn't "line 0: syntax error near unexpected token `>'..." mean bash
      > command is wrong?

      I'm using the statefulset from comment 0

      $ cat statefulset.yaml
      apiVersion: apps/v1
      kind: StatefulSet
      metadata:
      name: httpd
      spec:
      serviceName: "httpd"
      replicas: 1
      selector:
      matchLabels:
      app: httpd
      template:
      metadata:
      labels:
      app: httpd
      spec:
      containers:

      • name: httpd
        image: registry.redhat.io/rhel8/httpd-24:1-191
        ports:
      • containerPort: 80
        name: web
        lifecycle:
        postStart:
        exec:
        command:
      • /bin/sh
      • -c
      • curl -k https://<IP:PORT> > /tmp/urltest.txt

      — Additional comment from npinaeva@redhat.com on 2022-07-27 14:02:54 UTC —

      Did you replace <IP:PORT> here "curl -k https://<IP:PORT> > /tmp/urltest.txt"?

      — Additional comment from zzhao@redhat.com on 2022-07-28 07:43:25 UTC —

      (In reply to Nadia Pinaeva from comment #11)
      > Did you replace <IP:PORT> here "curl -k https://<IP:PORT> >
      > /tmp/urltest.txt"?

      oh my bad

      Tested again after replacing the ip and port with following


      apiVersion: apps/v1
      kind: StatefulSet
      metadata:
      name: httpd
      spec:
      serviceName: "httpd"
      replicas: 1
      selector:
      matchLabels:
      app: httpd
      template:
      metadata:
      labels:
      app: httpd
      spec:
      containers:

      • name: httpd
        image: registry.redhat.io/rhel8/httpd-24:1-191
        ports:
      • containerPort: 80
        name: web
        lifecycle:
        postStart:
        exec:
        command:
      • /bin/sh
      • -c
      • curl -k https://172.30.0.1:443 > /tmp/urltest.txt

      on 4.12.0-0.nightly-2022-07-27-133042

      $ oc get pod
      NAME READY STATUS RESTARTS AGE
      httpd-0 1/1 Running 0 2m28s

      — Additional comment from npinaeva@redhat.com on 2022-07-29 13:02:53 UTC —

      @swasthan@redhat.com yes, we are going to backport it to 4.10 (hopefully it will be faster than the fix itself )

            npinaeva@redhat.com Nadia Pinaeva
            npinaeva@redhat.com Nadia Pinaeva
            Anurag Saxena Anurag Saxena
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: