Upgrade from 4.11 to 4.12 is failing with Win32 internal error \"Access is denied\

    Upgrading from 4.11 windows-machine-config-operator-bundle-container-v6.0.4-3 to latest nightly is failing 

    Upgrading from:
4.11.57
windows-machine-config-operator-bundle-container-v6.0.4-3
     To:
4.12.0-0.nightly-2024-01-21-002245
windows-machine-config-operator-bundle-container-v7.2.1-6

    most likely

    1. Install v4.11 version 4.11.57 and operator windows-machine-config-operator-bundle-container-v6.0.4-3
    2. create 2 Windows machineset with  Windows Server 2019 Datacenter   10.0.17763.5329
    3. upgrade to 4.12.0-0.nightly-2024-01-21-002245 and windows-machine-config-operator-bundle-container-v7.2.1-6 operator      

    machine is stuck on Provisioned errors on wmco logs 
rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b   Provisioned   m5a.large    us-east-2   us-east-2a   50m
rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b   Running       m5a.large    us-east-2   us-east-2a   4h21m

{"level":"info","ts":"2024-01-23T14:28:33Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
{"level":"info","ts":"2024-01-23T14:28:33Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
{"level":"info","ts":"2024-01-23T14:28:33Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
{"level":"info","ts":"2024-01-23T14:28:44Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
{"level":"info","ts":"2024-01-23T14:28:44Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
{"level":"info","ts":"2024-01-23T14:28:44Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
{"level":"info","ts":"2024-01-23T14:30:35Z","logger":"controller.windowsmachine","msg":"processing","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","address":"10.0.147.93"}
{"level":"info","ts":"2024-01-23T14:31:08Z","logger":"wc 10.0.147.93","msg":"configuring"}
{"level":"error","ts":"2024-01-23T14:31:11Z","logger":"wc 10.0.147.93","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"","out":"Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":"2024-01-23T14:31:11Z","msg":"Reconciler error","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine","machine":{"name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","namespace":"openshift-machine-api"},"namespace":"openshift-machine-api","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","reconcileID":"0e5cb3e5-b554-45a4-a8fe-99176f65128f","error":"unable to configure instance i-0303b03467853ffb7: bootstrapping the Windows instance failed: error enabling Windows Containers feature: failed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n: error running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\": Process exited with status 1","errorVerbose":"Process exited with status 1\nerror running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:384\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nfailed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:921\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nerror enabling Windows Containers feature\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:576\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nbootstrapping the Windows instance failed\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:158\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nunable to configure instance i-0303b03467853ffb7\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:435\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":"2024-01-23T14:33:39Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
{"level":"info","ts":"2024-01-23T14:33:39Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
{"level":"info","ts":"2024-01-23T14:33:39Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
{"level":"info","ts":"2024-01-23T14:36:38Z","logger":"controller.windowsmachine","msg":"processing","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","address":"10.0.147.93"}
{"level":"info","ts":"2024-01-23T14:37:11Z","logger":"wc 10.0.147.93","msg":"configuring"}
{"level":"error","ts":"2024-01-23T14:37:14Z","logger":"wc 10.0.147.93","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"","out":"Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":"2024-01-23T14:37:14Z","msg":"Reconciler error","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine","machine":{"name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","namespace":"openshift-machine-api"},"namespace":"openshift-machine-api","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a-hsr2b","reconcileID":"b806143c-18d2-42d4-8e8d-0547b40736a2","error":"unable to configure instance i-0303b03467853ffb7: bootstrapping the Windows instance failed: error enabling Windows Containers feature: failed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n: error running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\": Process exited with status 1","errorVerbose":"Process exited with status 1\nerror running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Install-WindowsFeature -Name Containers\"\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:384\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:919\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nfailed to enable required Windows feature: Containers with output: Install-WindowsFeature : Win32 internal error \"Access is denied\" 0x5 occurred while reading the console output buffer. \r\nContact Microsoft Customer Support Services.\r\nAt line:1 char:1\r\n+ Install-WindowsFeature -Name Containers\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ReadError: (:) [Install-WindowsFeature], HostException\r\n    + FullyQualifiedErrorId : ReadConsoleOutput,Microsoft.Windows.ServerManager.Commands.AddWindowsFeatureCommand\r\n \r\n\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).enableContainersWindowsFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:921\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:575\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nerror enabling Windows Containers feature\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHostNameAndContainersFeature\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:576\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:466\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:156\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nbootstrapping the Windows instance failed\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:158\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:91\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:433\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nunable to configure instance i-0303b03467853ffb7\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:435\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:355\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":"2024-01-23T14:38:47Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
{"level":"info","ts":"2024-01-23T14:38:47Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
{"level":"info","ts":"2024-01-23T14:38:47Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
{"level":"info","ts":"2024-01-23T14:39:39Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b"}
{"level":"info","ts":"2024-01-23T14:39:39Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
{"level":"info","ts":"2024-01-23T14:39:39Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
{"level":"info","ts":"2024-01-23T14:43:52Z","logger":"controller.windowsmachine","msg":"deleting machine","windowsmachine":"openshift-machine-api/rrasouli
-1070-mm46d-windows-worker-us-east-2a-rws6b"}
{"level":"info","ts":"2024-01-23T14:43:52Z","logger":"controller.windowsmachine","msg":"unhealthy machine count for machineset","name":"rrasouli-1070-mm46d-windows-worker-us-east-2a","total":2,"unhealthy":1}
{"level":"info","ts":"2024-01-23T14:43:52Z","logger":"controller.windowsmachine","msg":"machine deletion restricted","windowsmachine":"openshift-machine-api/rrasouli-1070-mm46d-windows-worker-us-east-2a-rws6b","maxUnhealthyCount":1}
 

A working upgrade    

    wmco.log: https://privatebin.corp.redhat.com/?0f5141dbf95d4c12#6U6uqWqhui1iiKsroJGoSqqBc6vPah3RVRVYaD3UJ3dx