During an HCP volume scale test (80 HCPs per Management Cluster), we observed that it was unsuccessful because we ran into an AWS quota limit where there can only be a maximum of 200 security group rules per security group. This is because some NLBs created "kuberntes.io/rule/nlb/health" security group rules when they shouldn't have based on work done in HOSTEDCP-531

It's not clear how reproducible it is or how to reproduce it, but out of 80 HCPs it seemed to happen to 6 of them. I will attach a hypershift dump of one of the affected HCPs.