-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.10.z, 4.15.z
-
Moderate
-
No
-
CLOUD Sprint 246, CLOUD Sprint 247, CLOUD Sprint 248, CLOUD Sprint 249, CLOUD Sprint 250, CLOUD Sprint 251, CLOUD Sprint 252, CLOUD Sprint 253
-
8
-
Proposed
-
False
-
-
Customer Facing, Customer Reported
Description of problem:
On march 22 the autoscaler got in to a broken state with:
2023-03-22T12:46:49.148733289Z E0322 12:46:49.148726 1 static_autoscaler.go:364] Failed to fix node group sizes: failed to decrease MachineSet/openshift-machine-api/eu-3-compute-kgzn2-aro-machineset-compute-xl-germanywestcentral-1: attempt to delete existing nodes targetSize:4 delta:-1 existingNodes: 6
Accordingly to the code here :
Once it enters in the conditional of failure in the above IF... It won't recover to check for unschedulable pods present in this IF ELSE- function
UNTIL it gets fixed.
This can be seen below, where it stopped working at 2023-03-22T12:46 by not checking for pods that are not unschedulable and resumed just on March 23 at 19:14 UTC
So tldr; Once the quota issue was resolved (this was the specific error happening for the customer) between 19:05 ({}Last time it was seen a Quota Error{}) - 19:14, the autoscaler got fixed, and resumed to work.
I0323 19:15:21.925996 1 static_autoscaler.go:419] No unschedulable pods I0323 19:15:48.151241 1 static_autoscaler.go:419] No unschedulable pods I0323 19:14:00.640484 1 klogx.go:86] Pod eu-3-compute/compute-customer-27-7b6b98989-vrl7g is unschedulable I0323 19:14:31.471670 1 static_autoscaler.go:419] No unschedulable pods I0323 19:14:56.698237 1 static_autoscaler.go:419] No unschedulable pods I0322 12:46:23.716449 1 static_autoscaler.go:419] No unschedulable pods << last entry on March 22th I0322 12:45:08.041970 1 static_autoscaler.go:419] No unschedulable pods I0322 12:45:33.267542 1 static_autoscaler.go:419] No unschedulable pods I0322 12:45:58.492504 1 static_autoscaler.go:419] No unschedulable pods
Actual results:
Autoscaler does not work when gets into an error for a machine.
Expected results:
1. Entering an error for a machineautoscaler is OK. But the expectation here is that it will continuously work in case there are other healthy machineautoscalers in healthy mode. 2. Also, the error message is not helpful. While doing further testing after the error happened, the autoscaler was not being triggered, and the message does not says that it is expected to be fixed in order to be able to run again.
- is related to
-
OCPSTRAT-1106 Enable priority and least-waste expanders for cluster autoscaler
-
- In Progress
-
