Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2301

[gcp][CORS-1774] with "createFirewallRules: Enabled", after successful "create cluster" and then "destroy cluster", the created firewall-rules in the shared VPC are not deleted

    XMLWordPrintable

Details

    • Sprint 226
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      For {product-title} {product-version}, after you remove a cluster from a shared virtual private cloud (VPC), the {product-title} deletes any related firewall rules from the host and service projects.

      Previous versions of the {product-title} would only remove firewall rules from a project when the project's name matches the value of the `ProjectID` parameter in the `install-config.yaml` configuration file. This behavior might expose any remaining cluster resources in a project to security risks if the project reduces its security levels.

      (link:https://issues.redhat.com/browse/OCPBUGS-2301[*OCPBUGS-2301*])
      Show
      For {product-title} {product-version}, after you remove a cluster from a shared virtual private cloud (VPC), the {product-title} deletes any related firewall rules from the host and service projects. Previous versions of the {product-title} would only remove firewall rules from a project when the project's name matches the value of the `ProjectID` parameter in the `install-config.yaml` configuration file. This behavior might expose any remaining cluster resources in a project to security risks if the project reduces its security levels. (link: https://issues.redhat.com/browse/OCPBUGS-2301 [* OCPBUGS-2301 *])
    • Bug Fix
    • Done

    Description

      Description of problem:

      With "createFirewallRules: Enabled", after successful "create cluster" and then "destroy cluster", the created firewall-rules in the shared VPC are not deleted.

      Version-Release number of selected component (if applicable):

      $ ./openshift-install version
./openshift-install 4.12.0-0.nightly-2022-09-28-204419
built from commit 9eb0224926982cdd6cae53b872326292133e532d
release image registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc
release architecture amd64

      How reproducible:

      Always

      Steps to Reproduce:

      1. try IPI installation with "createFirewallRules: Enabled", which succeeded
2. try destroying the cluster, which succeeded
3. check firewall-rules in the shared VPC

      Actual results:

      After destroying the cluster, its firewall-rules created by installer in the shared VPC are not deleted.

      Expected results:

      Those firewall-rules should be deleted during destroying the cluster.

      Additional info:

      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules list --filter='network=installer-shared-vpc'
NAME                                NETWORK               DIRECTION  PRIORITY  ALLOW                                                    
                                                                                                 DENY  DISABLED
ci-op-xpn-ingress-common            installer-shared-vpc  INGRESS    60000     tcp:6443,tcp:22,tcp:80,tcp:443,icmp                      
                                                                                                       False
ci-op-xpn-ingress-health-checks     installer-shared-vpc  INGRESS    60000     tcp:30000-32767,udp:30000-32767,tcp:6080,tcp:6443,tcp:226
24,tcp:32335                                                                                           False
ci-op-xpn-ingress-internal-network  installer-shared-vpc  INGRESS    60000     udp:4789,udp:6081,udp:500,udp:4500,esp,tcp:9000-9999,udp:
9000-9999,tcp:10250,tcp:30000-32767,udp:30000-32767,tcp:10257,tcp:10259,tcp:22623,tcp:2379-2380        FalseTo show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.
$ 
$ yq-3.3.0 r test2/install-config.yaml platform
gcp:
  projectID: openshift-qe  
  region: us-central1
  computeSubnet: installer-shared-vpc-subnet-2
  controlPlaneSubnet: installer-shared-vpc-subnet-1
  createFirewallRules: Enabled
  network: installer-shared-vpc
  networkProjectID: openshift-qe-shared-vpc
$ 
$ yq-3.3.0 r test2/install-config.yaml metadata
creationTimestamp: null
name: jiwei-1013-01
$ 
$ openshift-install create cluster --dir test2
INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json"
INFO Consuming Install Config from target directory
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s (until 4:06AM) for the Kubernetes API at https://api.jiwei-1013-01.qe.gcp.devcluster.openshift.com:6443...
INFO API v1.24.0+8c7c967 up
INFO Waiting up to 30m0s (until 4:20AM) for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 40m0s (until 4:42AM) for the cluster at https://api.jiwei-1013-01.qe.gcp.devcluster.openshift.com:6443 to initialize...
INFO Checking to see if there is a route at openshift-console/console...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/fedora/test2/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.jiwei-1013-01.qe.gcp.devcluster.openshift.com
INFO Login to the console with user: "kubeadmin", and password: "wWPkc-8G2Lw-xe2Vw-DgWha"
INFO Time elapsed: 39m14s  
$ 
$ openshift-install destroy cluster --dir test2
INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json"
INFO Stopped instance jiwei-1013-01-464st-worker-b-pmg5z
INFO Stopped instance jiwei-1013-01-464st-worker-a-csg2j
INFO Stopped instance jiwei-1013-01-464st-master-1
INFO Stopped instance jiwei-1013-01-464st-master-2
INFO Stopped instance jiwei-1013-01-464st-master-0
INFO Deleted 2 recordset(s) in zone qe
INFO Deleted 3 recordset(s) in zone jiwei-1013-01-464st-private-zone
INFO Deleted DNS zone jiwei-1013-01-464st-private-zone
INFO Deleted bucket jiwei-1013-01-464st-image-registry-us-central1-ulgxgjfqxbdnrhd
INFO Deleted instance jiwei-1013-01-464st-master-0
INFO Deleted instance jiwei-1013-01-464st-worker-a-csg2j
INFO Deleted instance jiwei-1013-01-464st-master-1
INFO Deleted instance jiwei-1013-01-464st-worker-b-pmg5z
INFO Deleted instance jiwei-1013-01-464st-master-2
INFO Deleted disk jiwei-1013-01-464st-master-2
INFO Deleted disk jiwei-1013-01-464st-master-1
INFO Deleted disk jiwei-1013-01-464st-worker-b-pmg5z
INFO Deleted disk jiwei-1013-01-464st-master-0
INFO Deleted disk jiwei-1013-01-464st-worker-a-csg2j
INFO Deleted address jiwei-1013-01-464st-cluster-public-ip
INFO Deleted address jiwei-1013-01-464st-cluster-ip
INFO Deleted forwarding rule a516d89f9a4f14bdfb55a525b1a12a91
INFO Deleted forwarding rule jiwei-1013-01-464st-api
INFO Deleted forwarding rule jiwei-1013-01-464st-api-internal
INFO Deleted target pool a516d89f9a4f14bdfb55a525b1a12a91
INFO Deleted target pool jiwei-1013-01-464st-api
INFO Deleted backend service jiwei-1013-01-464st-api-internal
INFO Deleted instance group jiwei-1013-01-464st-master-us-central1-a
INFO Deleted instance group jiwei-1013-01-464st-master-us-central1-c
INFO Deleted instance group jiwei-1013-01-464st-master-us-central1-b
INFO Deleted health check jiwei-1013-01-464st-api-internal
INFO Deleted HTTP health check a516d89f9a4f14bdfb55a525b1a12a91
INFO Deleted HTTP health check jiwei-1013-01-464st-api
INFO Time elapsed: 4m18s   
$ 
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules list --filter='network=installer-shared-vpc'
NAME                                          NETWORK               DIRECTION  PRIORITY  ALLOW                                                                                                                                                     DENY  DISABLED
ci-op-xpn-ingress-common                      installer-shared-vpc  INGRESS    60000     tcp:6443,tcp:22,tcp:80,tcp:443,icmp                                                                                                                             False
ci-op-xpn-ingress-health-checks               installer-shared-vpc  INGRESS    60000     tcp:30000-32767,udp:30000-32767,tcp:6080,tcp:6443,tcp:22624,tcp:32335                                                                                           False
ci-op-xpn-ingress-internal-network            installer-shared-vpc  INGRESS    60000     udp:4789,udp:6081,udp:500,udp:4500,esp,tcp:9000-9999,udp:9000-9999,tcp:10250,tcp:30000-32767,udp:30000-32767,tcp:10257,tcp:10259,tcp:22623,tcp:2379-2380        False
jiwei-1013-01-464st-api                       installer-shared-vpc  INGRESS    1000      tcp:6443                                                                                                                                                        False
jiwei-1013-01-464st-control-plane             installer-shared-vpc  INGRESS    1000      tcp:22623,tcp:10257,tcp:10259                                                                                                                                   False
jiwei-1013-01-464st-etcd                      installer-shared-vpc  INGRESS    1000      tcp:2379-2380                                                                                                                                                   False
jiwei-1013-01-464st-health-checks             installer-shared-vpc  INGRESS    1000      tcp:6080,tcp:6443,tcp:22624                                                                                                                                     False
jiwei-1013-01-464st-internal-cluster          installer-shared-vpc  INGRESS    1000      tcp:30000-32767,udp:9000-9999,udp:30000-32767,udp:4789,udp:6081,tcp:9000-9999,udp:500,udp:4500,esp,tcp:10250                                                    False
jiwei-1013-01-464st-internal-network          installer-shared-vpc  INGRESS    1000      icmp,tcp:22                                                                                                                                                     False
k8s-a516d89f9a4f14bdfb55a525b1a12a91-http-hc  installer-shared-vpc  INGRESS    1000      tcp:30268                                                                                                                                                       False
k8s-fw-a516d89f9a4f14bdfb55a525b1a12a91       installer-shared-vpc  INGRESS    1000      tcp:80,tcp:443                                                                                                                                                  FalseTo show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.
$ 

FYI manually deleting those firewall-rules in the shared VPC does work.
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-api
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-api].
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-control-plane
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-control-plane].
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-etcd
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-etcd].
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-health-checks
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-health-checks].
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-internal-cluster
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-internal-cluster].
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-internal-network
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-internal-network].
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q k8s-a516d89f9a4f14bdfb55a525b1a12a91-http-hc
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/k8s-a516d89f9a4f14bdfb55a525b1a12a91-http-hc].
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q k8s-fw-a516d89f9a4f14bdfb55a525b1a12a91
Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-a516d89f9a4f14bdfb55a525b1a12a91].
$ 
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules list --filter='network=installer-shared-vpc'
NAME                                NETWORK               DIRECTION  PRIORITY  ALLOW                                                                                                                                                     DENY  DISABLED
ci-op-xpn-ingress-common            installer-shared-vpc  INGRESS    60000     tcp:6443,tcp:22,tcp:80,tcp:443,icmp                                                                                                                             False
ci-op-xpn-ingress-health-checks     installer-shared-vpc  INGRESS    60000     tcp:30000-32767,udp:30000-32767,tcp:6080,tcp:6443,tcp:22624,tcp:32335                                                                                           False
ci-op-xpn-ingress-internal-network  installer-shared-vpc  INGRESS    60000     udp:4789,udp:6081,udp:500,udp:4500,esp,tcp:9000-9999,udp:9000-9999,tcp:10250,tcp:30000-32767,udp:30000-32767,tcp:10257,tcp:10259,tcp:22623,tcp:2379-2380        FalseTo show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.
$ 

       

       

       

       

      Attachments

        Activity

          People

            rh-ee-bbarbach Brent Barbachem
            rhn-support-jiwei Jianli Wei
            Jianli Wei Jianli Wei
            Darragh Fitzmaurice Darragh Fitzmaurice
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: