Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-1721

[WMCO] WICD logging issue resolving PowerShell variable: could not resolve PowerShell variable ENDPOINT_IP: error running command with output New-HnsEndpoint

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • 4.19.0
    • 4.12, 4.19.0
    • Windows Containers
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 3
    • Moderate
    • None
    • None
    • None
    • WINC - Sprint 267, WINC - Sprint 268, WINC - Sprint 269, WINC - Sprint 270
    • 4
    • In Progress
    • Bug Fix
    • Hide
      *Cause*: Endpoint_IP variable not resolving
      *Consequence*: WICD logs are filled with Endpoint_IP errors
      *Fix*: added retries to be sure the Endpoint_IP was created before continuing
      *Result*: Bug doesn’t present anymore.
      Show
      *Cause*: Endpoint_IP variable not resolving *Consequence*: WICD logs are filled with Endpoint_IP errors *Fix*: added retries to be sure the Endpoint_IP was created before continuing *Result*: Bug doesn’t present anymore.
    • None
    • None
    • None
    • None

      Description of problem:

      When checking the WICD ERROR logs in WMCO 7.0.0 we can observe the following log appearing over and over:
      
      jfrancoa@localhost openshift-tests-private]$ oc adm node-logs -l kubernetes.io/os=windows --path=/wicd/windows-instance-config-daemon.exe.ERROR 
      ip-10-0-137-244.us-east-2.compute.internal Log file created at: 2022/09/26 06:17:29
      ip-10-0-137-244.us-east-2.compute.internal Running on machine: EC2AMAZ-114BK92
      ip-10-0-137-244.us-east-2.compute.internal Binary: Built with gc go1.18.1 for windows/amd64
      ip-10-0-137-244.us-east-2.compute.internal Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      ip-10-0-137-244.us-east-2.compute.internal E0926 06:17:29.790703    4200 controller.go:250] could not resolve PowerShell variable ENDPOINT_IP: error running command with output New-HnsEndpoint : Cannot process argument transformation on parameter 'NetworkId'. Cannot convert null to type 
      ip-10-0-137-244.us-east-2.compute.internal "System.Guid".
      ip-10-0-137-244.us-east-2.compute.internal At C:\Temp\network-conf.ps1:76 char:44
      ip-10-0-137-244.us-east-2.compute.internal +     $endpoint = New-HnsEndpoint -NetworkId $hns_network.ID -Name "VIP ...
      ip-10-0-137-244.us-east-2.compute.internal +                                            ~~~~~~~~~~~~~~~
      ip-10-0-137-244.us-east-2.compute.internal     + CategoryInfo          : InvalidData: (:) [New-HnsEndpoint], ParentContainsErrorRecordException
      ip-10-0-137-244.us-east-2.compute.internal     + FullyQualifiedErrorId : ParameterArgumentTransformationError,New-HnsEndpoint
      ip-10-0-137-244.us-east-2.compute.internal  
      ip-10-0-137-244.us-east-2.compute.internal : exit status 1
      ip-10-0-137-244.us-east-2.compute.internal E0926 06:17:32.204768    4200 controller.go:250] could not resolve PowerShell variable ENDPOINT_IP: error running command with output New-HnsEndpoint : Cannot process argument transformation on parameter 'NetworkId'. Cannot convert null to type 
      ip-10-0-137-244.us-east-2.compute.internal "System.Guid".
      ip-10-0-137-244.us-east-2.compute.internal At C:\Temp\network-conf.ps1:76 char:44
      
      Checking the data passed in the service, it looks like there is some issue resolving the ENDPOINT_IP variable passed in the powershellVariablesInCommand for the kube-proxy service:
      
      [jfrancoa@localhost openshift-tests-private]$ oc get cm windows-services-7.0.0-14c47b9 -n openshift-windows-machine-config-operator  -o yaml 
      apiVersion: v1
      data:
        files: '[]'
        services: '[{"name":"windows_exporter","path":"C:\\k\\windows_exporter.exe --collectors.enabled
          cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info","bootstrap":false,"priority":1},{"name":"hybrid-overlay-node","path":"C:\\k\\hybrid-overlay-node.exe
          --node NODE_NAME --k8s-kubeconfig C:\\k\\kubeconfig --windows-service --logfile
          C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log --loglevel 5","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"}],"dependencies":["kubelet"],"bootstrap":false,"priority":1},{"name":"kube-proxy","path":"C:\\k\\kube-proxy.exe
          --windows-service --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=NODE_NAME
          --kubeconfig=C:\\k\\kubeconfig --cluster-cidr=NODE_SUBNET --log-dir=C:\\var\\log\\kube-proxy\\
          --logtostderr=false --network-name=OVNKubernetesHybridOverlayNetwork --source-vip=ENDPOINT_IP
          --enable-dsr=false --v=4","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"},{"name":"NODE_SUBNET","nodeObjectJsonPath":"{.metadata.annotations.k8s\\.ovn\\.org/hybrid-overlay-node-subnet}"}],"powershellVariablesInCommand":[{"name":"ENDPOINT_IP","path":"C:\\Temp\\network-conf.ps1"}],"dependencies":["hybrid-overlay-node"],"bootstrap":false,"priority":2}]
      
      

      Version-Release number of selected component (if applicable):

      jfrancoa@localhost openshift-tests-private]$ oc version
      Client Version: 4.11.0-0.ci-2022-06-09-065118
      Kustomize Version: v4.5.4
      Server Version: 4.12.0-0.nightly-2022-09-25-071630
      Kubernetes Version: v1.24.0+8c7c967
      
      [jfrancoa@localhost openshift-tests-private]$ oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.12.0-0.nightly-2022-09-25-071630   True        False         10h     Error while reconciling 4.12.0-0.nightly-2022-09-25-071630: the cluster operator insights is not available
      
      [jfrancoa@localhost openshift-tests-private]$ oc get cm -n openshift-windows-machine-config-operator 
      NAME                                   DATA   AGE
      kube-root-ca.crt                       1      9h
      openshift-service-ca.crt               1      9h
      windows-machine-config-operator-lock   0      9h
      windows-services-7.0.0-14c47b9         2      9h
      
      

      How reproducible:

      Deploy a OSP4.12 cluster, install WMCO 7.0.0 and create two worker nodes. Verify that the log appears in windows-instance-config-daemon using the command: oc adm node-logs -l kubernetes.io/os=windows --path=/wicd/windows-instance-config-daemon.exe.ERROR 
      
      

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

      The WICD service has issue 
      

      Expected results:

      No variable resolving errors are printed out for the WICD service.
      

      Additional info:

      
      

              wgahnagl Skyler Clark
              rhn-engineering-jfrancoa Jose Luis Franco Arza (Inactive)
              None
              None
              Weinan Liu Weinan Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

                Created:
                Updated:
                Resolved: