[OCPBUGS-20525] Masters are not attached with the provided custom security groups which defined in platform.aws.defaultMachinePlatform

Type: Bug
Resolution: Done-Errata
Priority: Normal
Fix Version/s: 4.15.0
Affects Version/s: 4.14.0
Component/s: Installer / openshift-installer
Labels:
- triaged

Severity:
Moderate
Regression:
No
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Release Note Text:

Hide
Previously, if you only used the default machine configuration to specify existing AWS security groups when installing a cluster on AWS (`platform.aws.defaultMachinePlatform.additonalSecurityGroupsIDs`), the security groups were not applied to control plane machines. With this update, existing AWS security groups are correctly applied to control planes when they are specified using the default machine configuration. (link:https://issues.redhat.com/browse/OCPBUGS-20525[*~~OCPBUGS-20525~~*])

Show
Previously, if you only used the default machine configuration to specify existing AWS security groups when installing a cluster on AWS (`platform.aws.defaultMachinePlatform.additonalSecurityGroupsIDs`), the security groups were not applied to control plane machines. With this update, existing AWS security groups are correctly applied to control planes when they are specified using the default machine configuration. (link: https://issues.redhat.com/browse/OCPBUGS-20525 [* OCPBUGS-20525 *])
Release Note Type:
Bug Fix
Release Note Status:
Done
Target Version:

4.15.0

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:

Set custom security group IDs in the installconfig.platform.aws.defaultMachinePlatform.additionalSecurityGroupIDs field of install-config.yaml

such as: 

   apiVersion: v1
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     platform: {}
     replicas: 3
   compute:
   - architecture: amd64
     hyperthreading: Enabled
     name: worker
     platform: {}
     replicas: 3
   metadata:
     name: gpei-test1013
   platform:
     aws:
       region: us-east-2
       subnets:
       - subnet-0bc86b64e7736479c
       - subnet-0addd33c410b52251
       - subnet-093392f94a4099566
       - subnet-0b915a53042b6dc61
       defaultMachinePlatform:
         additionalSecurityGroupIDs:
         - sg-0fbc4c9733e6c18e7
         - sg-0b46b502b575d30ba
         - sg-02a59f8662d10c6d3


After installation, check the Security Groups attached to master and worker, master doesn't have the specified custom security groups attached while workers have. 

For one of the masters:
[root@preserve-gpei-worker k_files]# aws ec2 describe-instances --instance-ids i-08c0b0b6e4308be3b  --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json
[
    [
        [
            {
                "GroupName": "terraform-20231013000602175000000002",
                "GroupId": "sg-04b104d07075afe96"
            }
        ]
    ]
]

For one of the workers:
[root@preserve-gpei-worker k_files]# aws ec2 describe-instances --instance-ids i-00643f07748ec75da --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json
[
    [
        [
            {
                "GroupName": "test-sg2",
                "GroupId": "sg-0b46b502b575d30ba"
            },
            {
                "GroupName": "terraform-20231013000602174300000001",
                "GroupId": "sg-0d7cd50d4cb42e513"
            },
            {
                "GroupName": "test-sg3",
                "GroupId": "sg-02a59f8662d10c6d3"
            },
            {
                "GroupName": "test-sg1",
                "GroupId": "sg-0fbc4c9733e6c18e7"
            }
        ]
    ]
]


Also checked the master's controlplanemachineset, it does have the custom security groups configured, but they're not attached to the master instance in the end.

[root@preserve-gpei-worker k_files]# oc get controlplanemachineset -n openshift-machine-api cluster -o yaml |yq .spec.template.machines_v1beta1_machine_openshift_io.spec.providerSpec.value.securityGroups
- filters:
    - name: tag:Name
      values:
        - gpei-test1013-8lwtb-master-sg
- id: sg-02a59f8662d10c6d3
- id: sg-0b46b502b575d30ba
- id: sg-0fbc4c9733e6c18e7

Version-Release number of selected component (if applicable):

4.14.0-0.nightly-2023-10-12-104602

How reproducible:

Always

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

It works well when setting the security groups in installconfig.controlPlane.platform.aws.additionalSecurityGroupIDs

blocks

OCPBUGS-22771 Masters are not attached with the provided custom security groups which defined in platform.aws.defaultMachinePlatform

Closed

is cloned by

OCPBUGS-22771 Masters are not attached with the provided custom security groups which defined in platform.aws.defaultMachinePlatform

Closed

links to

openshift/installer#7589: OCPBUGS-20525: aws: use security groups from defaultMachinePlatform

RHEA-2023:7198 rpm

Errata Tool added a comment - 2024/02/27 8:53 PM

Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

For information on the advisory (Critical: OpenShift Container Platform 4.15.0 bug fix and security update), and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2023:7198

Errata Tool added a comment - 2024/02/27 8:53 PM Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Critical: OpenShift Container Platform 4.15.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:7198

Rafael Fonseca dos Santos added a comment - 2023/10/26 8:59 AM

gpei@redhat.com that's great to hear. I just suggested those scenarios because they were the ones I could have broken with my changes.

Rafael Fonseca dos Santos added a comment - 2023/10/26 8:59 AM gpei@redhat.com that's great to hear. I just suggested those scenarios because they were the ones I could have broken with my changes.

Gaoyun Pei added a comment - 2023/10/26 7:40 AM

rdossant Thanks for the suggestions! And sorry for the delay on verifying this issue, we're still working the final full functional testing for 4.14 GA. I made a try on verifying the scenarios you suggested, and I think all of them are working as expected.

> Only defaultMachinePlatform stanza, no additional SGs
> defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, no additional SGs

  controlPlane:
    architecture: amd64
    hyperthreading: Enabled
    name: master
    platform: {}
    replicas: 3
  compute:
  - architecture: amd64
    hyperthreading: Enabled
    name: worker
    platform: {}
    replicas: 3
  metadata:
    name: gpei-2604
  platform:
    aws:
      region: us-east-2

After Installation, check the SG attached to master, only the one created by installer

[root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-026e9a4ad6ecea8c9  --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json
[
    [
        [
            {
                "GroupName": "terraform-20231026063829893000000001",
                "GroupId": "sg-00b53dc59d84c2bdf"
            }
        ]
    ]
]

> Only defaultMachinePlatform stanza, additional SGs
> defaultMachinePlatform stanza, additional SGs; controlPlane stanza, no additional SGs

   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     platform: {}
     replicas: 3
   compute:
   - architecture: amd64
     hyperthreading: Enabled
     name: worker
     platform: {}
     replicas: 3
   metadata:
     name: gpei-2601
   platform:
     aws:
       region: us-east-2
       subnets:
       - subnet-0829a499fc7318cd1
       - subnet-08f08704cbe9aa291
       - subnet-03cb750a0b41b6869
       - subnet-0f67baf37e9ef3277
       defaultMachinePlatform:
         additionalSecurityGroupIDs:
         - sg-0ea6627c63944fd34
         - sg-0f2f8929a3d8a3f7a
         - sg-020a34693a093bdcc

After installation, check the SGs attached the master, they're the one defined in defaultMachinePlatform stanza.

[root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-0e18d4af87815ea2b --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json
[
    [
        [
            {
                "GroupName": "test-sg1",
                "GroupId": "sg-0ea6627c63944fd34"
            },
            {
                "GroupName": "test-sg2",
                "GroupId": "sg-0f2f8929a3d8a3f7a"
            },
            {
                "GroupName": "test-sg3",
                "GroupId": "sg-020a34693a093bdcc"
            },
            {
                "GroupName": "terraform-20231026002846713600000002",
                "GroupId": "sg-0ce9a94cb236f2bd6"
            }
        ]
    ]
]

> defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, additional SGs

   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     platform:
       aws:
         additionalSecurityGroupIDs:
         - sg-0c8a02fb5ef28a632
         - sg-021096647e5c6a7eb
     replicas: 3
   compute:
   - architecture: amd64
     hyperthreading: Enabled
     name: worker
     platform: {}
     replicas: 3
   metadata:
     name: gpei-2603
   platform:
     aws:
       region: us-east-2
       subnets:
       - subnet-0a352e768bc161090
       - subnet-02db5a4f8fa5a973f
       - subnet-0512b46e39b267acb
       - subnet-0211475b1f25e74bf

After installation, check the SGs attached the master, they're the one defined in controlPlane stanza.

[root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-07a506fa8b236a957 --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json
[
    [
        [
            {
                "GroupName": "gpei-2603-master-1",
                "GroupId": "sg-0c8a02fb5ef28a632"
            },
            {
                "GroupName": "gpei-2603-master-2",
                "GroupId": "sg-021096647e5c6a7eb"
            },
            {
                "GroupName": "terraform-20231026045342902300000001",
                "GroupId": "sg-02ef96aedd289a431"
            }
        ]
    ]
]

>defaultMachinePlatform stanza, additional SGs; controlPlane stanza, different additional SGs

The SGs defined in my install-config.yaml is like below:

  controlPlane:
    architecture: amd64
    hyperthreading: Enabled
    name: master
    platform:
      aws:
        additionalSecurityGroupIDs:
        - sg-0f2b093e570f7c34a
        - sg-0c52307d426ffc267
    replicas: 3
  compute:
  - architecture: amd64
    hyperthreading: Enabled
    name: worker
    platform: {}
    replicas: 3
  metadata:
    name: gpei-2602
  platform:
    aws:
      region: us-east-2
      subnets:
      - subnet-0719a234808cf2462
      - subnet-02c4e6a6c3b769af5
      - subnet-090d22a883fe60abc
      - subnet-020950dcbf528b660
      defaultMachinePlatform:
        additionalSecurityGroupIDs:
        - sg-0a211ec4acac87527
        - sg-090ab2e3122deeffb
        - sg-0b621262dcb3b0244

After installation, check the SGs attached the master, they're the one defined in controlPlane stanza.

[root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-0429ddc4c0b73b71d --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json
[
    [
        [
            {
                "GroupName": "gpei-2602-master-1",
                "GroupId": "sg-0f2b093e570f7c34a"
            },
            {
                "GroupName": "terraform-20231026045341918300000002",
                "GroupId": "sg-05a45d56afb900326"
            },
            {
                "GroupName": "gpei-2602-master-2",
                "GroupId": "sg-0c52307d426ffc267"
            }
        ]
    ]
]

Mark this issue as verified with 4.15.0-0.nightly-2023-10-26-011313

Gaoyun Pei added a comment - 2023/10/26 7:40 AM rdossant Thanks for the suggestions! And sorry for the delay on verifying this issue, we're still working the final full functional testing for 4.14 GA. I made a try on verifying the scenarios you suggested, and I think all of them are working as expected. > Only defaultMachinePlatform stanza, no additional SGs > defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, no additional SGs controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: {} replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 3 metadata: name: gpei-2604 platform: aws: region: us-east-2 After Installation, check the SG attached to master, only the one created by installer [root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-026e9a4ad6ecea8c9 --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json [ [ [ { "GroupName" : "terraform-20231026063829893000000001" , "GroupId" : "sg-00b53dc59d84c2bdf" } ] ] ] > Only defaultMachinePlatform stanza, additional SGs > defaultMachinePlatform stanza, additional SGs; controlPlane stanza, no additional SGs controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: {} replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 3 metadata: name: gpei-2601 platform: aws: region: us-east-2 subnets: - subnet-0829a499fc7318cd1 - subnet-08f08704cbe9aa291 - subnet-03cb750a0b41b6869 - subnet-0f67baf37e9ef3277 defaultMachinePlatform: additionalSecurityGroupIDs: - sg-0ea6627c63944fd34 - sg-0f2f8929a3d8a3f7a - sg-020a34693a093bdcc After installation, check the SGs attached the master, they're the one defined in defaultMachinePlatform stanza. [root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-0e18d4af87815ea2b --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json [ [ [ { "GroupName" : "test-sg1" , "GroupId" : "sg-0ea6627c63944fd34" }, { "GroupName" : "test-sg2" , "GroupId" : "sg-0f2f8929a3d8a3f7a" }, { "GroupName" : "test-sg3" , "GroupId" : "sg-020a34693a093bdcc" }, { "GroupName" : "terraform-20231026002846713600000002" , "GroupId" : "sg-0ce9a94cb236f2bd6" } ] ] ] > defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, additional SGs controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: aws: additionalSecurityGroupIDs: - sg-0c8a02fb5ef28a632 - sg-021096647e5c6a7eb replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 3 metadata: name: gpei-2603 platform: aws: region: us-east-2 subnets: - subnet-0a352e768bc161090 - subnet-02db5a4f8fa5a973f - subnet-0512b46e39b267acb - subnet-0211475b1f25e74bf After installation, check the SGs attached the master, they're the one defined in controlPlane stanza. [root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-07a506fa8b236a957 --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json [ [ [ { "GroupName" : "gpei-2603-master-1" , "GroupId" : "sg-0c8a02fb5ef28a632" }, { "GroupName" : "gpei-2603-master-2" , "GroupId" : "sg-021096647e5c6a7eb" }, { "GroupName" : "terraform-20231026045342902300000001" , "GroupId" : "sg-02ef96aedd289a431" } ] ] ] >defaultMachinePlatform stanza, additional SGs; controlPlane stanza, different additional SGs The SGs defined in my install-config.yaml is like below: controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: aws: additionalSecurityGroupIDs: - sg-0f2b093e570f7c34a - sg-0c52307d426ffc267 replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 3 metadata: name: gpei-2602 platform: aws: region: us-east-2 subnets: - subnet-0719a234808cf2462 - subnet-02c4e6a6c3b769af5 - subnet-090d22a883fe60abc - subnet-020950dcbf528b660 defaultMachinePlatform: additionalSecurityGroupIDs: - sg-0a211ec4acac87527 - sg-090ab2e3122deeffb - sg-0b621262dcb3b0244 After installation, check the SGs attached the master, they're the one defined in controlPlane stanza. [root@preserve-gpei-worker ~]# aws ec2 describe-instances --instance-ids i-0429ddc4c0b73b71d --query 'Reservations[*].Instances[*].SecurityGroups[*]' --output json [ [ [ { "GroupName" : "gpei-2602-master-1" , "GroupId" : "sg-0f2b093e570f7c34a" }, { "GroupName" : "terraform-20231026045341918300000002" , "GroupId" : "sg-05a45d56afb900326" }, { "GroupName" : "gpei-2602-master-2" , "GroupId" : "sg-0c52307d426ffc267" } ] ] ] Mark this issue as verified with 4.15.0-0.nightly-2023-10-26-011313

Rafael Fonseca dos Santos added a comment - 2023/10/18 6:37 PM

When validating the fix, it would be nice to include the following scenarios:

Only defaultMachinePlatform stanza, no additional SGs -> only the Installer created SGs are used (and nothing breaks)
Only defaultMachinePlatform stanza, additional SGs -> Installer created + additional SGs are used
defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, no additional SGs -> only Installer created SGs are used (and nothing breaks)
defaultMachinePlatform stanza, additional SGs; controlPlane stanza, no additional SGs -> Installer created + additional SGs used
defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, additional SGs -> Installer created + additional SGs used
defaultMachinePlatform stanza, additional SGs; controlPlane stanza, different additional SGs -> Installer created + controlPlane additional SGs used.

Rafael Fonseca dos Santos added a comment - 2023/10/18 6:37 PM When validating the fix, it would be nice to include the following scenarios: Only defaultMachinePlatform stanza, no additional SGs -> only the Installer created SGs are used (and nothing breaks) Only defaultMachinePlatform stanza, additional SGs -> Installer created + additional SGs are used defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, no additional SGs -> only Installer created SGs are used (and nothing breaks) defaultMachinePlatform stanza, additional SGs; controlPlane stanza, no additional SGs -> Installer created + additional SGs used defaultMachinePlatform stanza, no additional SGs; controlPlane stanza, additional SGs -> Installer created + additional SGs used defaultMachinePlatform stanza, additional SGs; controlPlane stanza, different additional SGs -> Installer created + controlPlane additional SGs used.

Assignee:: Rafael Fonseca dos Santos

Reporter:: Gaoyun Pei

QA Contact:: Gaoyun Pei

Doc Contact:: Mike Pytlak (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2023/10/13 3:29 AM

Updated:: 2024/02/27 8:53 PM

Resolved:: 2024/02/27 8:53 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Errata Tool added a comment - 2024/02/27 8:53 PM

Expand comment: Errata Tool added a comment - 2024/02/27 8:53 PM

Collapse comment: Rafael Fonseca dos Santos added a comment - 2023/10/26 8:59 AM

Expand comment: Rafael Fonseca dos Santos added a comment - 2023/10/26 8:59 AM

Collapse comment: Gaoyun Pei added a comment - 2023/10/26 7:40 AM

Expand comment: Gaoyun Pei added a comment - 2023/10/26 7:40 AM

Collapse comment: Rafael Fonseca dos Santos added a comment - 2023/10/18 6:37 PM

Expand comment: Rafael Fonseca dos Santos added a comment - 2023/10/18 6:37 PM

People

Dates