Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-5026

[Hypershift][4.12-Private cluster] Nodes/network/ingress not reaching to Running state

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 4.13.0, 4.12.0
    • HyperShift
    • None
    • None
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Nodes/ingress/network/dns not reaching to Running/Ready state

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-12-16-075715
      4.13.0-0.nightly-2022-12-17-092129

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create standalone OCP cluster
      2. Follow the steps as mentioned to create Private hypershift guest 
         cluster which has endpointAccess: Private   
      
         https://hypershift-docs.netlify.app/how-to/aws/deploy-aws-private- 
                clusters/ 

      Actual results:

      1. Nodes not readystate. 
      2. Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, openshift-samples, service-ca are not available

      Expected results:

      Nodes and operators should be up and Running. 

      Additional info:

      Private cluster never reached to Working state from past 2 weeks payloads. 
      [core@ip-10-0-142-253 ~]$ oc get nodes
      NAME                                         STATUS     ROLES    AGE     VERSION
      ip-10-0-130-89.us-east-2.compute.internal    NotReady   worker   9m47s   v1.25.2+0003605
      ip-10-0-135-234.us-east-2.compute.internal   NotReady   worker   9m35s   v1.25.2+0003605
      ip-10-0-142-253.us-east-2.compute.internal   NotReady   worker   9m47s   v1.25.2+0003605
      
      
      
      
      [core@ip-10-0-142-253 ~]$ oc describe nodes ip-10-0-130-89.us-east-2.compute.internal
      Conditions:
        Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
        ----             ------  -----------------                 ------------------                ------                       -------
        MemoryPressure   False   Sat, 17 Dec 2022 23:21:51 +0000   Sat, 17 Dec 2022 23:15:42 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
        DiskPressure     False   Sat, 17 Dec 2022 23:21:51 +0000   Sat, 17 Dec 2022 23:15:42 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
        PIDPressure      False   Sat, 17 Dec 2022 23:21:51 +0000   Sat, 17 Dec 2022 23:15:42 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
        Ready            False   Sat, 17 Dec 2022 23:21:51 +0000   Sat, 17 Dec 2022 23:15:42 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
      
      Events:
        Type     Reason                   Age                  From             Message
        ----     ------                   ----                 ----             -------
        Normal   Starting                 10m                  kubelet          Starting kubelet.
        Normal   NodeHasSufficientMemory  10m (x2 over 10m)    kubelet          Node ip-10-0-130-89.us-east-2.compute.internal status is now: NodeHasSufficientMemory
        Normal   NodeHasNoDiskPressure    10m (x2 over 10m)    kubelet          Node ip-10-0-130-89.us-east-2.compute.internal status is now: NodeHasNoDiskPressure
        Normal   NodeHasSufficientPID     10m (x2 over 10m)    kubelet          Node ip-10-0-130-89.us-east-2.compute.internal status is now: NodeHasSufficientPID
        Normal   NodeAllocatableEnforced  10m                  kubelet          Updated Node Allocatable limit across pods
        Normal   RegisteredNode           10m                  node-controller  Node ip-10-0-130-89.us-east-2.compute.internal event: Registered Node ip-10-0-130-89.us-east-2.compute.internal in Controller
        Warning  ErrorReconcilingNode     103s (x26 over 10m)  controlplane     [k8s.ovn.org/node-chassis-id annotation not found for node ip-10-0-130-89.us-east-2.compute.internal, macAddress annotation not found for node "ip-10-0-130-89.us-east-2.compute.internal" , k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-0-130-89.us-east-2.compute.internal"]
      
      
      
      rohitpatil@ropatil-mac privatecluster % oc get hostedclusters -A -o yaml       
      message: Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, openshift-samples, service-ca are not available
      
      rohitpatil@ropatil-mac privatecluster % oc get hostedclusters -A
      NAMESPACE   NAME            VERSION   KUBECONFIG                       PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
      clusters    413gstp164662             413gstp164662-admin-kubeconfig   Partial    True        False         The hosted control plane is available
      
      
      
      Mentioned the more messages here: https://issues.redhat.com/browse/OCPBUGS-4490

       

              agarcial@redhat.com Alberto Garcia Lamela
              ropatil@redhat.com Rohit Patil
              Jie Zhao Jie Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: