Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15988

BYOH node on AWS server 2022 reconciliation is failing: Unable to connect to the remote server\r\nAt line:1 char:1\r\n+ Invoke-RestMethod -UseBasicParsing -Uri http://169.254.169.254

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • 4.15.0
    • 4.13.z, 4.12.z, 4.14.z
    • Windows Containers
    • None
    • Critical
    • No
    • 3
    • WINC - Sprint 242
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: Routes to the metadata endpoint were added as non-persistent routes that were lost when the node is deconfigured or re-configured

      Consequence: Once a node is deconfigured or reconfigured, it can fails to get configured again as the metadata endpoint is unreachable

      Fix: WMCO runs the AWS EC2 launch v2 service after deconfiguration so that the routes are restored

      Result: VM can be configured into a node even after it is deconfigured
      Show
      Cause: Routes to the metadata endpoint were added as non-persistent routes that were lost when the node is deconfigured or re-configured Consequence: Once a node is deconfigured or reconfigured, it can fails to get configured again as the metadata endpoint is unreachable Fix: WMCO runs the AWS EC2 launch v2 service after deconfiguration so that the routes are restored Result: VM can be configured into a node even after it is deconfigured
    • Bug Fix

      Description of problem:

      BYOH node (configured with IP in windows-instances CM) bootstraping is failing on AWS server 2022 after reconciling the node:
      # oc annotate node <node name> --overwrite windowsmachineconfig.openshift.io/version=invalidVersion

      Version-Release number of selected component (if applicable):

      8.0.1-d0be5e8
      Server Version: 4.13.0-0.nightly-2023-07-09-175026
      Kubernetes Version: v1.26.6+a7ee68b
      

      How reproducible:

      100%

      Steps to Reproduce:

      1. install a BYOH node
      2. Configure CM with its IP address and username
      3. wait until the node added 
      4. reconcile the node by the command
      oc annotate node <node name> --overwrite windowsmachineconfig.openshift.io/version=invalidVersion

      Actual results:

      The BYOH node cannot get be added with server 2022 on AWS
      {"level":"info","ts":"2023-07-10T13:41:58Z","logger":"wc 10.0.141.40","msg":"transferring files"}
      {"level":"error","ts":"2023-07-10T13:42:18Z","logger":"wc 10.0.141.40","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe bootstrap --desired-version 8.0.1-d0be5e8 --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshift-windows-machine-config-operator\"","out":"I0710 13:42:12.532283    7344 bootstrap.go:60] bootstrapping Windows instance\nI0710 13:42:12.596008    7344 controller.go:277] created service containerd\nI0710 13:42:13.756983    7344 controller.go:324] updating service containerd\nI0710 13:42:19.241073    7344 controller.go:290] successfully reconciled service containerd\nI0710 13:42:19.241850    7344 controller.go:277] created service kubelet\nF0710 13:42:19.728106    7344 bootstrap.go:62] could not resolve PowerShell variable HOSTNAME_OVERRIDE: error running command with output Invoke-RestMethod : Unable to connect to the remote server\r\nAt line:1 char:1\r\n+ Invoke-RestMethod -UseBasicParsing -Uri http://169.254.169.254/latest ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc \r\n   eption\r\n    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand\r\n \r\n: exit status 1\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:373\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:442\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:159\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:84\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).ensureInstancesAreUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:314\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileNodes\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:279\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:189\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
      {"level":"info","ts":"2023-07-10T13:42:18Z","logger":"wc 10.0.141.40","msg":"failed to bootstrap node","command":"C:\\k\\windows-instance-config-daemon.exe bootstrap --desired-version 8.0.1-d0be5e8 --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshift-windows-machine-config-operator","output":"I0710 13:42:12.532283    7344 bootstrap.go:60] bootstrapping Windows instance\nI0710 13:42:12.596008    7344 controller.go:277] created service containerd\nI0710 13:42:13.756983    7344 controller.go:324] updating service containerd\nI0710 13:42:19.241073    7344 controller.go:290] successfully reconciled service containerd\nI0710 13:42:19.241850    7344 controller.go:277] created service kubelet\nF0710 13:42:19.728106    7344 bootstrap.go:62] could not resolve PowerShell variable HOSTNAME_OVERRIDE: error running command with output Invoke-RestMethod : Unable to connect to the remote server\r\nAt line:1 char:1\r\n+ Invoke-RestMethod -UseBasicParsing -Uri http://169.254.169.254/latest ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc \r\n   eption\r\n    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand\r\n \r\n: exit status 1\n"}
      {"level":"error","ts":"2023-07-10T13:42:18Z","msg":"Reconciler error","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","ConfigMap":{"name":"windows-instances","namespace":"openshift-windows-machine-config-operator"},"namespace":"openshift-windows-machine-config-operator","name":"windows-instances","reconcileID":"ff4cf89c-e889-44bf-8313-728bad1a7af9","error":"error configuring host with address 10.0.141.40: bootstrapping the Windows instance failed: error running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe bootstrap --desired-version 8.0.1-d0be5e8 --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshift-windows-machine-config-operator\": Process exited with status 1","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}

      Expected results:

      BYOH node reconcilled after version annotation on server 2022

      Additional info:

      oc get cm -n openshift-windows-machine-config-operator windows-services-8.0.1-d0be5e8 -o yaml
      apiVersion: v1
      data:
        files: '[]'
        services: '[{"name":"containerd","path":"C:\\k\\containerd\\containerd.exe --config
          C:\\k\\containerd\\containerd_conf.toml --log-file C:\\var\\log\\containerd\\containerd.log
          --run-service --log-level info","powershellPreScripts":[{"path":"C:\\Temp\\windows-defender-exclusion.ps1
          -BinPath C:\\k\\containerd\\containerd.exe"}],"bootstrap":true,"priority":0},{"name":"kubelet","path":"C:\\k\\kube-log-runner.exe
          -log-file=C:\\var\\log\\kubelet\\kubelet.log C:\\k\\kubelet.exe --config=C:\\k\\kubelet.conf
          --bootstrap-kubeconfig=C:\\k\\bootstrap-kubeconfig --kubeconfig=C:\\k\\kubeconfig
          --cert-dir=c:\\var\\lib\\kubelet\\pki\\ --windows-service --register-with-taints=os=Windows:NoSchedule
          --node-labels=node.openshift.io/os_id=Windows --container-runtime=remote --container-runtime-endpoint=npipe://./pipe/containerd-containerd
          --resolv-conf= --windows-priorityclass=ABOVE_NORMAL_PRIORITY_CLASS --v=2 --cloud-provider=aws
          --hostname-override=HOSTNAME_OVERRIDE","powershellPreScripts":[{"variableName":"HOSTNAME_OVERRIDE","path":"Invoke-RestMethod
          -UseBasicParsing -Uri http://169.254.169.254/latest/meta-data/local-hostname"}],"dependencies":["containerd"],"bootstrap":true,"priority":1},{"name":"windows_exporter","path":"C:\\k\\windows_exporter.exe
          --collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info","bootstrap":false,"priority":2},{"name":"hybrid-overlay-node","path":"C:\\k\\hybrid-overlay-node.exe
          --node NODE_NAME --k8s-kubeconfig C:\\k\\kubeconfig --windows-service --logfile
          C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"}],"dependencies":["kubelet"],"bootstrap":false,"priority":2},{"name":"kube-proxy","path":"C:\\k\\kube-log-runner.exe
          -log-file=C:\\var\\log\\kube-proxy\\kube-proxy.log C:\\k\\kube-proxy.exe --windows-service
          --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=NODE_NAME
          --kubeconfig=C:\\k\\kubeconfig --cluster-cidr=NODE_SUBNET --network-name=OVNKubernetesHybridOverlayNetwork
          --source-vip=ENDPOINT_IP --enable-dsr=false --v=2","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"},{"name":"NODE_SUBNET","nodeObjectJsonPath":"{.metadata.annotations.k8s\\.ovn\\.org/hybrid-overlay-node-subnet}"}],"powershellPreScripts":[{"variableName":"ENDPOINT_IP","path":"C:\\Temp\\network-conf.ps1"}],"dependencies":["hybrid-overlay-node"],"bootstrap":false,"priority":3}]'
      immutable: true
      kind: ConfigMap
      metadata:
        creationTimestamp: "2023-07-10T06:27:44Z"
        name: windows-services-8.0.1-d0be5e8
        namespace: openshift-windows-machine-config-operator
        resourceVersion: "32807"
        uid: 3d3a0796-3601-4908-b0c4-781f7ac7410a

        1. MachineSet.yaml
          2 kB
          Aravindh Puthiyaparambil
        2. wmco.log
          122 kB
          Aravindh Puthiyaparambil

              paravindh Aravindh Puthiyaparambil (Inactive)
              rrasouli Aharon Rasouli
              Aharon Rasouli Aharon Rasouli
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: