Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-27775

Upgrade from 4.11 to 4.12 is failing with Win32 internal error \"Access is denied\

XMLWordPrintable

    • No
    • 3
    • WINC - Sprint 250
    • 1
    • False
    • Hide

      upgrades aren't working

      Show
      upgrades aren't working
    • Hide
      *Cause*: upgrade path included earlier released version
      *Consequence*: causing the operator to fail while upgrading to older patch versions
      *Fix*: add the olm.skipRange to avoid older patch versions during upgrade
      *Result*: upgrade from 6.x to latest 7.x succeed
      Show
      *Cause*: upgrade path included earlier released version *Consequence*: causing the operator to fail while upgrading to older patch versions *Fix*: add the olm.skipRange to avoid older patch versions during upgrade *Result*: upgrade from 6.x to latest 7.x succeed
    • Bug Fix

      Description of problem:

          Upgrading from 4.11 windows-machine-config-operator-bundle-container-v6.0.4-3 to latest nightly is failing 

      Version-Release number of selected component (if applicable):

          Upgrading from:
      4.11.57
      windows-machine-config-operator-bundle-container-v6.0.4-3
           To:
      4.12.0-0.nightly-2024-01-21-002245
      windows-machine-config-operator-bundle-container-v7.2.1-6

      How reproducible:

          most likely

      Steps to Reproduce:

          1. Install v4.11 version 4.11.57 and operator windows-machine-config-operator-bundle-container-v6.0.4-3
          2. create 2 Windows machineset with  Windows Server 2019 Datacenter   10.0.17763.5329
          3. upgrade to 4.12.0-0.nightly-2024-01-21-002245 and windows-machine-config-operator-bundle-container-v7.2.1-6 operator      

      Actual results:

          machine is stuck on Provisioned errors on wmco logs 
      rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b   Provisioned   m5a.large    us-east-2   us-east-2a   50m
      rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b   Running       m5a.large    us-east-2   us-east-2a   4h21m
      
      {"level":"info","ts":"2024-01-23T14:28:33Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
      {"level":"info","ts":"2024-01-23T14:28:33Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
      {"level":"info","ts":"2024-01-23T14:28:33Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
      {"level":"info","ts":"2024-01-23T14:28:44Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
      {"level":"info","ts":"2024-01-23T14:28:44Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
      {"level":"info","ts":"2024-01-23T14:28:44Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
      {"level":"info","ts":"2024-01-23T14:30:35Z","logger":"controller.windowsmachine","msg":"processing","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","address":"10.0.147.93"}
      {"level":"info","ts":"2024-01-23T14:31:08Z","logger":"wc 10.0.147.93","msg":"configuring"}
      {"level":"error","ts":"2024-01-23T14:31:11Z","logger":"wc 10.0.147.93","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"","out":"Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
      {"level":"error","ts":"2024-01-23T14:31:11Z","msg":"Reconciler error","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine","machine":{"name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","namespace":"openshift-machine-api"},"namespace":"openshift-machine-api","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","reconcileID":"0e5cb3e5-b554-45a4-a8fe-99176f65128f","error":"unable to configure instance i-0303b03467853ffb7: bootstrapping the Windows instance failed: error enabling Windows Containers feature: failed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n: error running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\": Process exited with status 1","errorVerbose":"Process exited with status 1\nerror running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:384\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nfailed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:921\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nerror enabling Windows Containers feature\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:576\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nbootstrapping the Windows instance failed\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:158\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nunable to configure instance i-0303b03467853ffb7\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:435\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
      {"level":"info","ts":"2024-01-23T14:33:39Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
      {"level":"info","ts":"2024-01-23T14:33:39Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
      {"level":"info","ts":"2024-01-23T14:33:39Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
      {"level":"info","ts":"2024-01-23T14:36:38Z","logger":"controller.windowsmachine","msg":"processing","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","address":"10.0.147.93"}
      {"level":"info","ts":"2024-01-23T14:37:11Z","logger":"wc 10.0.147.93","msg":"configuring"}
      {"level":"error","ts":"2024-01-23T14:37:14Z","logger":"wc 10.0.147.93","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"","out":"Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
      {"level":"error","ts":"2024-01-23T14:37:14Z","msg":"Reconciler error","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine","machine":{"name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","namespace":"openshift-machine-api"},"namespace":"openshift-machine-api","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","reconcileID":"b806143c-18d2-42d4-8e8d-0547b40736a2","error":"unable to configure instance i-0303b03467853ffb7: bootstrapping the Windows instance failed: error enabling Windows Containers feature: failed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n: error running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\": Process exited with status 1","errorVerbose":"Process exited with status 1\nerror running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:384\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nfailed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:921\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nerror enabling Windows Containers feature\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:576\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nbootstrapping the Windows instance failed\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:158\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nunable to configure instance i-0303b03467853ffb7\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:435\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
      {"level":"info","ts":"2024-01-23T14:38:47Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
      {"level":"info","ts":"2024-01-23T14:38:47Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
      {"level":"info","ts":"2024-01-23T14:38:47Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
      {"level":"info","ts":"2024-01-23T14:39:39Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
      {"level":"info","ts":"2024-01-23T14:39:39Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
      {"level":"info","ts":"2024-01-23T14:39:39Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
      {"level":"info","ts":"2024-01-23T14:43:52Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli
      -1070-mm46d-windows-worker-us-east-2a-rws6b"}
      {"level":"info","ts":"2024-01-23T14:43:52Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
      {"level":"info","ts":"2024-01-23T14:43:52Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
       

      Expected results:

      A working upgrade    

      Additional info:

          wmco.log: https://privatebin.corp.redhat.com/?0f5141dbf95d4c12#6U6uqWqhui1iiKsroJGoSqqBc6vPah3RVRVYaD3UJ3dx

            jvaldes@redhat.com Jose Valdes
            rrasouli Aharon Rasouli
            Aharon Rasouli Aharon Rasouli
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: