Submariner gateway node: Error updating load balancer with new hosts map

Name:         submariner
Namespace:    acm-aws-nmanos-a2
Labels:       <none>
Annotations:  <none>
API Version:  submarineraddon.open-cluster-management.io/v1alpha1
Kind:         SubmarinerConfig
Metadata:
  Creation Timestamp:  2022-12-06T06:03:55Z
  Finalizers:
    submarineraddon.open-cluster-management.io/config-cleanup
  Generation:  2
  Managed Fields:
    API Version:  submarineraddon.open-cluster-management.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:IPSecIKEPort:
        f:IPSecNATTPort:
        f:NATTDiscoveryPort:
        f:NATTEnable:
        f:cableDriver:
        f:credentialsSecret:
          .:
          f:name:
        f:gatewayConfig:
          .:
          f:aws:
            .:
            f:instanceType:
          f:gateways:
        f:imagePullSpecs:
        f:loadBalancerEnable:
        f:subscriptionConfig:
          .:
          f:channel:
          f:source:
          f:sourceNamespace:
          f:startingCSV:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-12-06T06:03:55Z
    API Version:  submarineraddon.open-cluster-management.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"submarineraddon.open-cluster-management.io/config-cleanup":
      f:spec:
        f:gatewayConfig:
          f:azure:
            .:
            f:instanceType:
          f:gcp:
            .:
            f:instanceType:
          f:rhos:
            .:
            f:instanceType:
    Manager:      submariner
    Operation:    Update
    Time:         2022-12-06T06:03:55Z
    API Version:  submarineraddon.open-cluster-management.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:managedClusterInfo:
          .:
          f:clusterName:
          f:infraId:
          f:platform:
          f:region:
          f:vendor:
    Manager:         submariner
    Operation:       Update
    Subresource:     status
    Time:            2022-12-06T06:04:00Z
  Resource Version:  112556
  UID:               80e4f944-e3ef-45a1-ab5c-37a37dc4d72c
Spec:
  IP Sec IKE Port:      502
  IP Sec NATT Port:     4502
  NATT Discovery Port:  4900
  NATT Enable:          true
  Cable Driver:         libreswan
  Credentials Secret:
    Name:  acm-aws-nmanos-a2-aws-creds
  Gateway Config:
    Aws:
      Instance Type:  c5d.large
    Azure:
      Instance Type:  Standard_D4s_v3
    Gateways:         1
    Gcp:
      Instance Type:  n1-standard-4
    Rhos:
      Instance Type:  PnTAE.CPU_16_Memory_32768_Disk_80
  Image Pull Specs:
  Load Balancer Enable:  false
  Subscription Config:
    Channel:           stable-0.12
    Source:            submariner-stable-0-12-catalog
    Source Namespace:  submariner-operator
    Starting CSV:      submariner.v0.12
Status:
  Conditions:
    Last Transition Time:  2022-12-06T06:04:00Z
    Message:               Submariner cluster environment was prepared
    Reason:                SubmarinerClusterEnvPrepared
    Status:                True
    Type:                  SubmarinerClusterEnvironmentPrepared
    Last Transition Time:  2022-12-06T06:04:01Z
    Message:               SubmarinerConfig was applied
    Reason:                SubmarinerConfigApplied
    Status:                True
    Type:                  SubmarinerConfigApplied
    Last Transition Time:  2022-12-06T06:08:02Z
    Message:               1 node(s) ("ip-10-166-10-129.us-west-1.compute.internal") are labeled as gateways
    Reason:                Success
    Status:                True
    Type:                  SubmarinerGatewaysLabeled
  Managed Cluster Info:
    Cluster Name:  acm-aws-nmanos-a2
    Infra Id:      aws-nmanos-a2-h79vd
    Platform:      AWS
    Region:        us-west-1
    Vendor:        OpenShift
Events:            <none>

NAME                                           STATUS   ROLES    AGE    VERSION            INTERNAL-IP      EXTERNAL-IP     OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-166-10-129.us-west-1.compute.internal    Ready    worker   21m    v1.23.12+8a6bfe4   10.166.10.129    18.144.167.96   Red Hat Enterprise Linux CoreOS 410.84.202211282118-0 (Ootpa)   4.18.0-305.65.1.el8_4.x86_64   cri-o://1.23.3-20.rhaos4.10.git89344de.el8
ip-10-166-154-70.us-west-1.compute.internal    Ready    worker   161m   v1.23.12+8a6bfe4   10.166.154.70    <none>          Red Hat Enterprise Linux CoreOS 410.84.202211282118-0 (Ootpa)   4.18.0-305.65.1.el8_4.x86_64   cri-o://1.23.3-20.rhaos4.10.git89344de.el8
ip-10-166-180-183.us-west-1.compute.internal   Ready    worker   164m   v1.23.12+8a6bfe4   10.166.180.183   <none>          Red Hat Enterprise Linux CoreOS 410.84.202211282118-0 (Ootpa)   4.18.0-305.65.1.el8_4.x86_64   cri-o://1.23.3-20.rhaos4.10.git89344de.el8
ip-10-166-255-43.us-west-1.compute.internal    Ready    worker   161m   v1.23.12+8a6bfe4   10.166.255.43    <none>          Red Hat Enterprise Linux CoreOS 410.84.202211282118-0 (Ootpa)   4.18.0-305.65.1.el8_4.x86_64   cri-o://1.23.3-20.rhaos4.10.git89344de.el8

No resources found in submariner-operator namespace.

UpdateLoadBalancerFailed                           Error updating load balancer with new hosts map[ip-10-166-10-129.us-west-1.compute.internal:{} ip-10-166-129-205.us-west-1.compute.internal:{} ip-10-166-150-70.us-west-1.compute.internal:{} ip-10-166-154-70.us-west-1.compute.internal:{} ip-10-166-180-183.us-west-1.compute.internal:{} ip-10-166-198-122.us-west-1.compute.internal:{} ip-10-166-255-43.us-west-1.compute.internal:{}]: Multiple tagged security groups found for instance i-0e722eeedf0d69eeb; ensure only the k8s security group is tagged; the tagged groups were sg-07f460ad239364d9b(aws-nmanos-a2-h79vd-submariner-gw-sg) sg-0f8d89b4202126713(terraform-20221206033036145600000002)