-
Bug
-
Resolution: Done
-
Normal
-
None
-
4.10
-
None
-
Moderate
-
None
-
False
-
+++ This bug was initially created as a clone of Bug #2081562 +++
Description of problem:
lifecycle.posStart does not have network connectivity on OpenShiftSDN CNI. (OVNKubernetes does not have the issue)
Version-Release number of selected component (if applicable):
4.10
How reproducible:
always
Steps to Reproduce:
1. create statefulset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ oc create -f statefulset.yaml
$ cat statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: httpd
spec:
serviceName: "httpd"
replicas: 1
selector:
matchLabels:
app: httpd
template:
metadata:
labels:
app: httpd
spec:
containers:
- name: httpd
image: registry.redhat.io/rhel8/httpd-24:1-191
ports: - containerPort: 80
name: web
lifecycle:
postStart:
exec:
command: - /bin/sh
- -c
- curl -k https://<IP:PORT> > /tmp/urltest.txt
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Actual results:
PostStartHook fails
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
36s Normal Killing pod/httpd-0 FailedPostStartHook
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expected results:
PostStartHook should not fail.
Additional info:
by adding a dummy initContainers, you can workaround the issue.
something like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
spec:
initContainers:
- name: init-myservice
image: busybox:1.28
command: ['sh', '-c', 'sleep 2']
containers: - name: httpd
image: registry.redhat.io/rhel8/httpd-24:1-191
ports: - containerPort: 80
name: web
lifecycle:
postStart:
exec:
command: - /bin/sh
- -c
- curl -k <IP:PORT> > /tmp/urltest.txt
....
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
— Additional comment from rphillips@redhat.com on 2022-05-11 19:48:10 UTC —
crio's contract with networking is to have networking up when the container starts. Moving to the openshift-sdn team to help triage what is going on.
— Additional comment from hyoskim@redhat.com on 2022-06-09 00:40:33 UTC —
Hello,
Is there any update on this issue?
— Additional comment from npinaeva@redhat.com on 2022-06-09 07:53:38 UTC —
Hello, yeah we found the root cause and working on the fix now - PR should be ready by the end of the week
— Additional comment from aos-team-art-private@bot.bugzilla.redhat.com on 2022-07-24 15:21:48 UTC —
Elliott changed bug status from MODIFIED to ON_QA.
This bug is expected to ship in the next 4.11 release.
— Additional comment from errata-xmlrpc@redhat.com on 2022-07-27 00:18:40 UTC —
This bug has been added to advisory RHSA-2022:5069 by OpenShift Release Team Bot (ocp-build/buildvm.openshift.eng.bos.redhat.com@REDHAT.COM)
— Additional comment from swasthan@redhat.com on 2022-07-27 05:38:55 UTC —
Hello Team, thank you for the help so far!
May we know if this is going to backport in v4.10.z as well?
Regards,
Swadeep
— Additional comment from zzhao@redhat.com on 2022-07-27 06:40:30 UTC —
this fixed PR is merged to build 4.12.0-0.nightly-2022-07-24-180529
So I update the target version to 4.12 version.
— Additional comment from zzhao@redhat.com on 2022-07-27 06:48:33 UTC —
still failed on build 4.12.0-0.nightly-2022-07-26-131732
Creating above statefulset and pod still cannot be worked with same error
27s Warning FailedPostStartHook pod/httpd-0 Exec lifecycle hook ([/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt]) for Container "httpd" in Pod "httpd-0_default(7e519841-7092-4513-928b-03c7783ddc7d)" failed - error: command '/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt' exited with 1: /bin/sh: -c: line 0: syntax error near unexpected token `>'...
85s Normal Killing pod/httpd-0 FailedPostStartHook
— Additional comment from npinaeva@redhat.com on 2022-07-27 12:50:53 UTC —
Hey @zzhao@redhat.com can you share full statefulset yaml you're running?
Doesn't "line 0: syntax error near unexpected token `>'..." mean bash command is wrong?
— Additional comment from zzhao@redhat.com on 2022-07-27 13:36:43 UTC —
(In reply to Nadia Pinaeva from comment #9)
> Hey @zzhao@redhat.com can you share full statefulset yaml you're running?
> Doesn't "line 0: syntax error near unexpected token `>'..." mean bash
> command is wrong?
I'm using the statefulset from comment 0
$ cat statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: httpd
spec:
serviceName: "httpd"
replicas: 1
selector:
matchLabels:
app: httpd
template:
metadata:
labels:
app: httpd
spec:
containers:
- name: httpd
image: registry.redhat.io/rhel8/httpd-24:1-191
ports: - containerPort: 80
name: web
lifecycle:
postStart:
exec:
command: - /bin/sh
- -c
- curl -k https://<IP:PORT> > /tmp/urltest.txt
— Additional comment from npinaeva@redhat.com on 2022-07-27 14:02:54 UTC —
Did you replace <IP:PORT> here "curl -k https://<IP:PORT> > /tmp/urltest.txt"?
— Additional comment from zzhao@redhat.com on 2022-07-28 07:43:25 UTC —
(In reply to Nadia Pinaeva from comment #11)
> Did you replace <IP:PORT> here "curl -k https://<IP:PORT> >
> /tmp/urltest.txt"?
oh my bad
Tested again after replacing the ip and port with following
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: httpd
spec:
serviceName: "httpd"
replicas: 1
selector:
matchLabels:
app: httpd
template:
metadata:
labels:
app: httpd
spec:
containers:
- name: httpd
image: registry.redhat.io/rhel8/httpd-24:1-191
ports: - containerPort: 80
name: web
lifecycle:
postStart:
exec:
command: - /bin/sh
- -c
- curl -k https://172.30.0.1:443 > /tmp/urltest.txt
—
on 4.12.0-0.nightly-2022-07-27-133042
$ oc get pod
NAME READY STATUS RESTARTS AGE
httpd-0 1/1 Running 0 2m28s
— Additional comment from npinaeva@redhat.com on 2022-07-29 13:02:53 UTC —
@swasthan@redhat.com yes, we are going to backport it to 4.10 (hopefully it will be faster than the fix itself )
- is cloned by
-
OCPBUGS-255 lifecycle.posStart hook does not have network connectivity.
- Closed