Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-6040

ovnkube node pod crashed after converting to a dual-stack cluster network

XMLWordPrintable

    • Critical
    • None
    • SDN Sprint 231, SDN Sprint 232
    • 2
    • Rejected
    • False
    • Hide

      None

      Show
      None

      This is a clone of issue OCPBUGS-4411. The following is the description of the original issue:

      Description of problem:

      manually configure ipv6 addresses and route on ipv4 OCP cluster to create a dualstack cluster, newly created pods will stay in 'ContainerCreating' status

      Version-Release number of selected component (if applicable):

      4.12

      How reproducible:

      Steps to Reproduce:

      1. enable ipv6 in network.
      # more patch_dual.yaml 
      - op: add
        path: /spec/clusterNetwork/-
        value:
          cidr: fd01::/48
          hostPrefix: 64
      - op: add
        path: /spec/serviceNetwork/-
        value: fd02::/112
      # oc patch network.config.openshift.io cluster --type='json' --patch-file patch_dual.yaml
       
      2. Configure ipv6 addresses and routes
      
      PODS=$(oc get pods -n openshift-cluster-node-tuning-operator -l openshift-app=tuned --field-selector=status.phase=Running --no-headers -o name)
      i=10
      for pod in $PODS; do
        oc exec -n openshift-cluster-node-tuning-operator $pod -- ip -6 addr add fd00:172:22::${i}/64 dev br-ex
        oc exec -n openshift-cluster-node-tuning-operator $pod -- ip -6 route add default via fd00:172:22::1 dev br-ex
        ((i=i+1))
      done 
      
      3. create pods and they will stay in ContainerCreating status.
      
      4. if remove the ipv6 configuration in network, newly created pods can be ready.
      
      
      

      Actual results:

      Pod can not be running

      Expected results:

      Pod should be ready with both ipv4 and ipv6 address.

      Additional info:

      version:
      # oc version
      Client Version: 4.12.0-0.nightly-2022-11-30-182550
      Kustomize Version: v4.5.7
      Server Version: 4.12.0-0.nightly-2022-11-30-182550
      Kubernetes Version: v1.25.2+5533733
      
      Describe pods:
      # oc describe pod iperf-rc-normal-qg6zd 
      Name:             iperf-rc-normal-qg6zd
      Namespace:        offload-testing
      Priority:         0
      Service Account:  default
      Node:             openshift-qe-025.lab.eng.rdu2.redhat.com/192.168.111.54
      Start Time:       Thu, 01 Dec 2022 21:35:28 -0500
      Labels:           name=iperf-pods-normal
      Annotations:      k8s.ovn.org/pod-networks:
                          {"default":{"ip_addresses":["10.129.2.7/23","fd01:0:0:6::3/64"],"mac_address":"0a:58:0a:81:02:07","gateway_ips":["10.129.2.1","fd01:0:0:6:...
                        openshift.io/scc: restricted-v2
                        seccomp.security.alpha.kubernetes.io/pod: runtime/default
      Status:           Pending
      IP:               
      IPs:              <none>
      Controlled By:    ReplicationController/iperf-rc-normal
      Containers:
        iperf:
          Container ID:   
          Image:          quay.io/openshifttest/iperf3@sha256:440c59251338e9fcf0a00d822878862038d3b2e2403c67c940c7781297953614
          Image ID:       
          Port:           <none>
          Host Port:      <none>
          State:          Waiting
            Reason:       ContainerCreating
          Ready:          False
          Restart Count:  0
          Limits:
            memory:  340Mi
          Requests:
            memory:     340Mi
          Environment:  <none>
          Mounts:
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4266b (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        kube-api-access-4266b:
          Type:                    Projected (a volume that contains injected data from multiple sources)
          TokenExpirationSeconds:  3607
          ConfigMapName:           kube-root-ca.crt
          ConfigMapOptional:       <nil>
          DownwardAPI:             true
          ConfigMapName:           openshift-service-ca.crt
          ConfigMapOptional:       <nil>
      QoS Class:                   Burstable
      Node-Selectors:              <none>
      Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                                   node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
      Events:
        Type     Reason                  Age                     From     Message
        ----     ------                  ----                    ----     -------
        Warning  FailedCreatePodSandBox  3m4s (x173 over 5h50m)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_iperf-rc-normal-qg6zd_offload-testing_18673f13-37b4-40ea-aa5d-85654dfa5c85_0(4899f7150492fa4cd895c62d0ec25ac5c1507016037c31b6019849083b42cdb5): error adding pod offload-testing_iperf-rc-normal-qg6zd to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [offload-testing/iperf-rc-normal-qg6zd/18673f13-37b4-40ea-aa5d-85654dfa5c85:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[offload-testing/iperf-rc-normal-qg6zd 4899f7150492fa4cd895c62d0ec25ac5c1507016037c31b6019849083b42cdb5] [offload-testing/iperf-rc-normal-qg6zd 4899f7150492fa4cd895c62d0ec25ac5c1507016037c31b6019849083b42cdb5] failed to configure pod interface: timed out waiting for OVS port binding (ovn-installed) for 0a:58:0a:81:02:07 [10.129.2.7/23 fd01:0:0:6::3/64]
      '

       

            jcaamano@redhat.com Jaime Caamaño Ruiz
            openshift-crt-jira-prow OpenShift Prow Bot
            Ross Brattain Ross Brattain
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: