Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-13244

Instance configurations fails on Windows Server 2019 without the container feature

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • 4.14.0
    • 4.14.0
    • Windows Containers
    • None
    • No
    • 3
    • WINC - Sprint 237
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: WMCO fails to configure Windows Server 2019 instances on Azure that do not have the Containers feature pre-installed with "Install-WindowsFeature : Win32 internal error "Access is denied" 0x5
      occurred while reading the console output buffer". This happens because the command results in a progress bar being displayed and that cannot
      be displayed over ssh and results in the error.

      Consequence: VM never get configured into a node

      Fix: A fix was introduced in WMCO to silence the progress bar

      Result: VM gets configured into a node
      Show
      Cause: WMCO fails to configure Windows Server 2019 instances on Azure that do not have the Containers feature pre-installed with "Install-WindowsFeature : Win32 internal error "Access is denied" 0x5 occurred while reading the console output buffer". This happens because the command results in a progress bar being displayed and that cannot be displayed over ssh and results in the error. Consequence: VM never get configured into a node Fix: A fix was introduced in WMCO to silence the progress bar Result: VM gets configured into a node
    • Bug Fix

      Description of problem

      WMCO fails to configure Windows Server 2019 instances on Azure that do not have the Containers feature pre-installed with the following error:

      2023-05-05T18:11:02Z	ERROR	wc 10.0.128.7	error running	{"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"", "out": "Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n", "error": "Process exited with status 1"}
      github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run
      	/build/windows-machine-config-operator/pkg/windows/windows.go:373
      github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature
      	/build/windows-machine-config-operator/pkg/windows/windows.go:893
      github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature
      	/build/windows-machine-config-operator/pkg/windows/windows.go:530
      github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap
      	/build/windows-machine-config-operator/pkg/windows/windows.go:430
      github.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure
      	/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:159
      github.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate
      	/build/windows-machine-config-operator/controllers/controllers.go:84
      github.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine
      	/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433
      github.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile
      	/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:354
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
      	/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
      
      How reproducible:{code:none}
      Always
      

      Steps to Reproduce:

      1. Bring up a OCP 4.14 cluster on Azure
      2. Create a MachineSet with sku: 2019-datacenter-smalldisk
      3. Launch WMCO from source
      

      Actual results:

      Node fails to get configured
      

      Expected results:

      Node should be configured
      

      Additional info

      The issue here is that Install-WindowsFeature -Name Containers results in a progress bar being displayed and causes the Access is denied error. However I noticed that if you manually installed the containers feature while WMCO was retrying the configuration, the install still does not go through and times out waiting for a hybrid-ovn annotation. In the hybrid-overlay logs I saw:

      F0508 19:28:55.018792    5108 hybrid-overlay-node.go:53] this version of Windows does not support HostRoute policies; network communication between this node and its pods will not work. HostRoute
      policies are available as a KB update for Windows Server 2019 version 1809 and out of the box in Windows Server 2019 version 1903.
      

      Interestingly 2019-datacenter-smalldisk and 2019-datacenter-with-containers-smalldisk have identical patch and hotfixes

      PS C:\Users\capi> [System.Environment]::OSVersion.Version
      
      Major  Minor  Build  Revision
      -----  -----  -----  --------
      10     0      17763  0
      
      PS C:\Users\capi> Get-HotFix
      
      Source        Description      HotFixID      InstalledBy          InstalledOn               
      ------        -----------      --------      -----------          -----------
      win2-mjrss    Update           KB5022511                          4/5/2023 12:00:00 AM
      win2-mjrss    Security Update  KB5025229                          4/5/2023 12:00:00 AM
      win2-mjrss    Security Update  KB5023789                          4/5/2023 12:00:00 AM
      
      

      Note

      It is unclear why the progress bar issue does not show up with 2022 images which do not have the Container feature installed.

            paravindh Aravindh Puthiyaparambil
            paravindh Aravindh Puthiyaparambil
            Aharon Rasouli Aharon Rasouli
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: