Uploaded image for project: 'OpenShift Hosted Control Plane'
  1. OpenShift Hosted Control Plane
  2. HOSTEDCP-1419

Always include default security group in ROSA clusters

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Major Major
    • openshift-4.16
    • None
    • None
    • None
    • False
    • None
    • False
    • XCMSTRAT-320 - ROSA HCP: Additional Security Group(s) on Machine Pools
    • Hypershift Sprint 249
    • 0
    • 0
    • 0

      Currently the behavior is to only use the default security group if no security groups were specified for a NodePool. This makes it difficult to implement additional security groups in ROSA because there is no way to know the default security group on cluster creation. By always appending the default security group, any security groups specified on the NodePool become additional security groups.

            [HOSTEDCP-1419] Always include default security group in ROSA clusters

            He Liu added a comment - - edited

            Test result passed using the cli built by the latest hypershift codes and HO:

            $ hypershift -v
hypershift version openshift/hypershift: 5e2737fb73fb1c0415423d4e1f11fef1dbb9fb8e. Latest supported OCP: 4.16.0

# HO version:
$ oc logs -n hypershift -lapp=operator --tail=-1 -c operator | head -1 | jq
{
  "level": "info",
  "ts": "2024-02-21T08:11:46Z",
  "logger": "setup",
  "msg": "Starting hypershift-operator-manager",
  "version": "openshift/hypershift: 5e2737fb73fb1c0415423d4e1f11fef1dbb9fb8e. Latest supported OCP: 4.16.0"
} 

$ oc get hc -A
NAMESPACE   NAME        VERSION                              KUBECONFIG                   PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
clusters    heli-test   4.16.0-0.nightly-2024-02-21-020511   heli-test-admin-kubeconfig   Completed   True        False         The hosted control plane is available

            test steps:

            # check hc default sg:
$ oc get hc heli-test  -ojsonpath='{.status.platform.aws.defaultWorkerSecurityGroupID}' -n clusters
sg-070f580b3d2d90226

# check the default nodepool sg spec, we expected no sg spec now
$ oc get np heli-test-us-east-2a  -n clusters -ojsonpath='{.spec.platform}' | jq
{
  "aws": {
    "instanceProfile": "heli-test-27zhg-worker",
    "instanceType": "m5.large",
    "rootVolume": {
      "size": 120,
      "type": "gp3"
    },
    "subnet": {
      "id": "subnet-0c0937964e7b9c48d"
    }
  },
  "type": "AWS"
}

# check ec2 instances sg of this nodepool, expect the default sg is there
$ oc get awsmachinetemplate -n clusters-heli-test -ojsonpath='{.items[?(@.metadata.annotations.hypershift\.openshift\.io/nodePool=="clusters/heli-test-us-east-2a")].spec.template.spec.additionalSecurityGroups}' | jq
[
  {
    "id": "sg-070f580b3d2d90226"
  }
]

$ aws ec2 describe-instances --filter="Name=tag:Name,Values=heli-test-us-east-2a*" | jq -r '.Reservations[].Instances[].SecurityGroups'
[]
[
  {
    "GroupName": "heli-test-27zhg-default-sg",
    "GroupId": "sg-070f580b3d2d90226"
  }
]

# scale up default np replicas
$ oc scale np -n clusters heli-test-us-east-2a --replicas=2
nodepool.hypershift.openshift.io/heli-test-us-east-2a scaled

$ aws ec2 describe-instances --filter="Name=tag:Name,Values=heli-test-us-east-2a*" | jq -r '.Reservations[].Instances[].SecurityGroups'
[
  {
    "GroupName": "heli-test-27zhg-default-sg",
    "GroupId": "sg-070f580b3d2d90226"
  }
]
[
  {
    "GroupName": "heli-test-27zhg-default-sg",
    "GroupId": "sg-070f580b3d2d90226"
  }
]

1. 
# check the new node's sg
# create a new sg without 
hypershift create nodepool aws --name=heli-test --cluster-name=heli-test --node-count=1 --release-image=registry.ci.openshift.org/ocp/release:4.16.0-0.nightly-2024-02-21-020511 --namespace=clusters

# check the new sg spec and ec2 instance sg
# no sg spec
$ oc get np heli-test  -n clusters -ojsonpath='{.spec.platform}' | jq
{
  "aws": {
    "instanceProfile": "heli-test-27zhg-worker",
    "instanceType": "m5.large",
    "rootVolume": {
      "size": 120,
      "type": "gp3"
    },
    "subnet": {
      "id": "subnet-0c0937964e7b9c48d"
    }
  },
  "type": "AWS"
} 

# ec2 instance sg info
$  oc get awsmachine -n clusters-heli-test heli-test-6130f4e5-c9rss  -ojsonpath='{.spec.additionalSecurityGroups}' | jq
[
  {
    "id": "sg-070f580b3d2d90226"
  }
]

2. create a new nodepool with a new sg
# create a new sg in the vpc
$ aws ec2 create-security-group --group-name heli-test  --description "heli-test" --vpc-id vpc-02cd7c0ada194fc2c
{
    "GroupId": "sg-067f2cc0fea688d8d"
}

$ hypershift create nodepool aws --name=heli-test-2 --cluster-name=heli-test --node-count=1 --release-image=registry.ci.openshift.org/ocp/release:4.16.0-0.nightly-2024-02-21-020511 --namespace=clusters --securitygroup-id=sg-067f2cc0fea688d8d
NodePool heli-test-2 created

# check sg info

$ oc get np heli-test-2 -n clusters -ojsonpath='{.spec.platform.aws.securityGroups}' | jq
[
  {
    "id": "sg-067f2cc0fea688d8d"
  }
]

$ oc get awsmachinetemplate -n clusters-heli-test -ojsonpath='{.items[?(@.metadata.annotations.hypershift\.openshift\.io/nodePool=="clusters/heli-test-2")].spec.template.spec.additionalSecurityGroups}' | jq
[
  {
    "id": "sg-067f2cc0fea688d8d"
  },
  {
    "id": "sg-070f580b3d2d90226"
  }
]
 
$ aws ec2 describe-instances --filter="Name=tag:Name,Values=heli-test-2*" | jq -r '.Reservations[].Instances[].SecurityGroups'
[
  {
    "GroupName": "heli-test-27zhg-default-sg",
    "GroupId": "sg-070f580b3d2d90226"
  },
  {
    "GroupName": "heli-test",
    "GroupId": "sg-067f2cc0fea688d8d"
  }
]

            np condition:

            the default nodepool

             

            $ oc get np heli-test-us-east-2a -n clusters -ojsonpath='{.status.conditions[?(@.type=="AWSSecurityGroupAvailable")]}'| jq
{
  "lastTransitionTime": "2024-02-21T06:40:59Z",
  "message": "NodePool has a default security group",
  "observedGeneration": 4,
  "reason": "AsExpected",
  "status": "True",
  "type": "AWSSecurityGroupAvailable"
}

            the new nodepool with a new sg specified:

            $ oc get np heli-test-2 -n clusters -ojsonpath='{.status.conditions[?(@.type=="AWSSecurityGroupAvailable")]}'| jq
{
  "lastTransitionTime": "2024-02-21T08:05:13Z",
  "message": "NodePool has a default security group",
  "observedGeneration": 1,
  "reason": "AsExpected",
  "status": "True",
  "type": "AWSSecurityGroupAvailable"
}

            cewong@redhat.com Is it proper to show "NodePool has a default security group" even it has a new sg as well as a default sg?

            He Liu added a comment - - edited Test result passed using the cli built by the latest hypershift codes and HO: $ hypershift -v hypershift version openshift/hypershift: 5e2737fb73fb1c0415423d4e1f11fef1dbb9fb8e. Latest supported OCP: 4.16.0 # HO version: $ oc logs -n hypershift -lapp= operator --tail=-1 -c operator | head -1 | jq {   "level" : "info" ,   "ts" : "2024-02-21T08:11:46Z" ,   "logger" : "setup" ,   "msg" : "Starting hypershift- operator -manager" ,   "version" : "openshift/hypershift: 5e2737fb73fb1c0415423d4e1f11fef1dbb9fb8e. Latest supported OCP: 4.16.0" } $ oc get hc -A NAMESPACE   NAME        VERSION                              KUBECONFIG                   PROGRESS    AVAILABLE   PROGRESSING   MESSAGE clusters    heli-test   4.16.0-0.nightly-2024-02-21-020511   heli-test-admin-kubeconfig   Completed   True        False         The hosted control plane is available test steps: # check hc default sg: $ oc get hc heli-test  -ojsonpath= '{.status.platform.aws.defaultWorkerSecurityGroupID}' -n clusters sg-070f580b3d2d90226 # check the default nodepool sg spec, we expected no sg spec now $ oc get np heli-test-us-east-2a  -n clusters -ojsonpath= '{.spec.platform}' | jq {   "aws" : {     "instanceProfile" : "heli-test-27zhg-worker" ,     "instanceType" : "m5.large" ,     "rootVolume" : {       "size" : 120,       "type" : "gp3"     },     "subnet" : {       "id" : "subnet-0c0937964e7b9c48d"     }   },   "type" : "AWS" } # check ec2 instances sg of this nodepool, expect the default sg is there $ oc get awsmachinetemplate -n clusters-heli-test -ojsonpath= '{.items[?(@.metadata.annotations.hypershift\.openshift\.io/nodePool== "clusters/heli-test-us-east-2a" )].spec.template.spec.additionalSecurityGroups}' | jq [   {     "id" : "sg-070f580b3d2d90226"   } ] $ aws ec2 describe-instances --filter= "Name=tag:Name,Values=heli-test-us-east-2a*" | jq -r '.Reservations[].Instances[].SecurityGroups' [] [   {     "GroupName" : "heli-test-27zhg- default -sg" ,     "GroupId" : "sg-070f580b3d2d90226"   } ] # scale up default np replicas $ oc scale np -n clusters heli-test-us-east-2a --replicas=2 nodepool.hypershift.openshift.io/heli-test-us-east-2a scaled $ aws ec2 describe-instances --filter= "Name=tag:Name,Values=heli-test-us-east-2a*" | jq -r '.Reservations[].Instances[].SecurityGroups' [   {     "GroupName" : "heli-test-27zhg- default -sg" ,     "GroupId" : "sg-070f580b3d2d90226"   } ] [   {     "GroupName" : "heli-test-27zhg- default -sg" ,     "GroupId" : "sg-070f580b3d2d90226"   } ] 1. # check the new node's sg # create a new sg without hypershift create nodepool aws --name=heli-test --cluster-name=heli-test --node-count=1 --release-image=registry.ci.openshift.org/ocp/release:4.16.0-0.nightly-2024-02-21-020511 --namespace=clusters # check the new sg spec and ec2 instance sg # no sg spec $ oc get np heli-test  -n clusters -ojsonpath= '{.spec.platform}' | jq {   "aws" : {     "instanceProfile" : "heli-test-27zhg-worker" ,     "instanceType" : "m5.large" ,     "rootVolume" : {       "size" : 120,       "type" : "gp3"     },     "subnet" : {       "id" : "subnet-0c0937964e7b9c48d"     }   },   "type" : "AWS" } # ec2 instance sg info $  oc get awsmachine -n clusters-heli-test heli-test-6130f4e5-c9rss  -ojsonpath= '{.spec.additionalSecurityGroups}' | jq [   {     "id" : "sg-070f580b3d2d90226"   } ] 2. create a new nodepool with a new sg # create a new sg in the vpc $ aws ec2 create-security-group --group-name heli-test  --description "heli-test" --vpc-id vpc-02cd7c0ada194fc2c {     "GroupId" : "sg-067f2cc0fea688d8d" } $ hypershift create nodepool aws --name=heli-test-2 --cluster-name=heli-test --node-count=1 --release-image=registry.ci.openshift.org/ocp/release:4.16.0-0.nightly-2024-02-21-020511 --namespace=clusters --securitygroup-id=sg-067f2cc0fea688d8d NodePool heli-test-2 created # check sg info $ oc get np heli-test-2 -n clusters -ojsonpath= '{.spec.platform.aws.securityGroups}' | jq [   {     "id" : "sg-067f2cc0fea688d8d"   } ] $ oc get awsmachinetemplate -n clusters-heli-test -ojsonpath= '{.items[?(@.metadata.annotations.hypershift\.openshift\.io/nodePool== "clusters/heli-test-2" )].spec.template.spec.additionalSecurityGroups}' | jq [   {     "id" : "sg-067f2cc0fea688d8d"   },   {     "id" : "sg-070f580b3d2d90226"   } ] $ aws ec2 describe-instances --filter= "Name=tag:Name,Values=heli-test-2*" | jq -r '.Reservations[].Instances[].SecurityGroups' [   {     "GroupName" : "heli-test-27zhg- default -sg" ,     "GroupId" : "sg-070f580b3d2d90226"   },   {     "GroupName" : "heli-test" ,     "GroupId" : "sg-067f2cc0fea688d8d"   } ] np condition: the default nodepool   $ oc get np heli-test-us-east-2a -n clusters -ojsonpath= '{.status.conditions[?(@.type== "AWSSecurityGroupAvailable" )]}' | jq {   "lastTransitionTime" : "2024-02-21T06:40:59Z" ,   "message" : "NodePool has a default security group" ,   "observedGeneration" : 4,   "reason" : "AsExpected" ,   "status" : "True" ,   "type" : "AWSSecurityGroupAvailable" } the new nodepool with a new sg specified: $ oc get np heli-test-2 -n clusters -ojsonpath= '{.status.conditions[?(@.type== "AWSSecurityGroupAvailable" )]}' | jq {   "lastTransitionTime" : "2024-02-21T08:05:13Z" ,   "message" : "NodePool has a default security group" ,   "observedGeneration" : 1,   "reason" : "AsExpected" ,   "status" : "True" ,   "type" : "AWSSecurityGroupAvailable" } cewong@redhat.com Is it proper to show "NodePool has a default security group" even it has a new sg as well as a default sg?

            Irit Goihman (Inactive) added a comment -

            Thanks!

            Irit Goihman (Inactive) added a comment - Thanks!

            Irit Goihman (Inactive) added a comment -

            cewong@redhat.com any chance this change could be under an epic by itself and linked to the parent https://issues.redhat.com/browse/XCMSTRAT-320 ?
            This way we can track work by separate teams. Thanks!

            Irit Goihman (Inactive) added a comment - cewong@redhat.com any chance this change could be under an epic by itself and linked to the parent https://issues.redhat.com/browse/XCMSTRAT-320 ? This way we can track work by separate teams. Thanks!

              cewong@redhat.com Cesar Wong
              cewong@redhat.com Cesar Wong
              He Liu He Liu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: