Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2301

[gcp][CORS-1774] with "createFirewallRules: Enabled", after successful "create cluster" and then "destroy cluster", the created firewall-rules in the shared VPC are not deleted

    XMLWordPrintable

Details

    • Sprint 226
    • Rejected
    • Hide

      None

      Show
      None
    • Hide
      For {product-title} {product-version}, after you remove a cluster from a shared virtual private cloud (VPC), the {product-title} deletes any related firewall rules from the host and service projects.

      Previous versions of the {product-title} would only remove firewall rules from a project when the project's name matches the value of the `ProjectID` parameter in the `install-config.yaml` configuration file. This behavior might expose any remaining cluster resources in a project to security risks if the project reduces its security levels.

      (link:https://issues.redhat.com/browse/OCPBUGS-2301[*OCPBUGS-2301*])
      Show
      For {product-title} {product-version}, after you remove a cluster from a shared virtual private cloud (VPC), the {product-title} deletes any related firewall rules from the host and service projects. Previous versions of the {product-title} would only remove firewall rules from a project when the project's name matches the value of the `ProjectID` parameter in the `install-config.yaml` configuration file. This behavior might expose any remaining cluster resources in a project to security risks if the project reduces its security levels. (link: https://issues.redhat.com/browse/OCPBUGS-2301 [* OCPBUGS-2301 *])
    • Bug Fix
    • Done

    Description

      Description of problem:

      With "createFirewallRules: Enabled", after successful "create cluster" and then "destroy cluster", the created firewall-rules in the shared VPC are not deleted.

      Version-Release number of selected component (if applicable):

      $ ./openshift-install version
      ./openshift-install 4.12.0-0.nightly-2022-09-28-204419
      built from commit 9eb0224926982cdd6cae53b872326292133e532d
      release image registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc
      release architecture amd64
      

      How reproducible:

      Always

      Steps to Reproduce:

      1. try IPI installation with "createFirewallRules: Enabled", which succeeded
      2. try destroying the cluster, which succeeded
      3. check firewall-rules in the shared VPC 

      Actual results:

      After destroying the cluster, its firewall-rules created by installer in the shared VPC are not deleted.

      Expected results:

      Those firewall-rules should be deleted during destroying the cluster.

      Additional info:

      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules list --filter='network=installer-shared-vpc'
      NAME                                NETWORK               DIRECTION  PRIORITY  ALLOW                                                    
                                                                                                       DENY  DISABLED
      ci-op-xpn-ingress-common            installer-shared-vpc  INGRESS    60000     tcp:6443,tcp:22,tcp:80,tcp:443,icmp                      
                                                                                                             False
      ci-op-xpn-ingress-health-checks     installer-shared-vpc  INGRESS    60000     tcp:30000-32767,udp:30000-32767,tcp:6080,tcp:6443,tcp:226
      24,tcp:32335                                                                                           False
      ci-op-xpn-ingress-internal-network  installer-shared-vpc  INGRESS    60000     udp:4789,udp:6081,udp:500,udp:4500,esp,tcp:9000-9999,udp:
      9000-9999,tcp:10250,tcp:30000-32767,udp:30000-32767,tcp:10257,tcp:10259,tcp:22623,tcp:2379-2380        FalseTo show all fields of the firewall, please show in JSON format: --format=json
      To show all fields in table format, please see the examples in --help.
      $ 
      $ yq-3.3.0 r test2/install-config.yaml platform
      gcp:
        projectID: openshift-qe  
        region: us-central1
        computeSubnet: installer-shared-vpc-subnet-2
        controlPlaneSubnet: installer-shared-vpc-subnet-1
        createFirewallRules: Enabled
        network: installer-shared-vpc
        networkProjectID: openshift-qe-shared-vpc
      $ 
      $ yq-3.3.0 r test2/install-config.yaml metadata
      creationTimestamp: null
      name: jiwei-1013-01
      $ 
      $ openshift-install create cluster --dir test2
      INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json"
      INFO Consuming Install Config from target directory
      INFO Creating infrastructure resources...
      INFO Waiting up to 20m0s (until 4:06AM) for the Kubernetes API at https://api.jiwei-1013-01.qe.gcp.devcluster.openshift.com:6443...
      INFO API v1.24.0+8c7c967 up
      INFO Waiting up to 30m0s (until 4:20AM) for bootstrapping to complete...
      INFO Destroying the bootstrap resources...
      INFO Waiting up to 40m0s (until 4:42AM) for the cluster at https://api.jiwei-1013-01.qe.gcp.devcluster.openshift.com:6443 to initialize...
      INFO Checking to see if there is a route at openshift-console/console...
      INFO Install complete!
      INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/fedora/test2/auth/kubeconfig'
      INFO Access the OpenShift web-console here: https://console-openshift-console.apps.jiwei-1013-01.qe.gcp.devcluster.openshift.com
      INFO Login to the console with user: "kubeadmin", and password: "wWPkc-8G2Lw-xe2Vw-DgWha"
      INFO Time elapsed: 39m14s  
      $ 
      $ openshift-install destroy cluster --dir test2
      INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json"
      INFO Stopped instance jiwei-1013-01-464st-worker-b-pmg5z
      INFO Stopped instance jiwei-1013-01-464st-worker-a-csg2j
      INFO Stopped instance jiwei-1013-01-464st-master-1
      INFO Stopped instance jiwei-1013-01-464st-master-2
      INFO Stopped instance jiwei-1013-01-464st-master-0
      INFO Deleted 2 recordset(s) in zone qe
      INFO Deleted 3 recordset(s) in zone jiwei-1013-01-464st-private-zone
      INFO Deleted DNS zone jiwei-1013-01-464st-private-zone
      INFO Deleted bucket jiwei-1013-01-464st-image-registry-us-central1-ulgxgjfqxbdnrhd
      INFO Deleted instance jiwei-1013-01-464st-master-0
      INFO Deleted instance jiwei-1013-01-464st-worker-a-csg2j
      INFO Deleted instance jiwei-1013-01-464st-master-1
      INFO Deleted instance jiwei-1013-01-464st-worker-b-pmg5z
      INFO Deleted instance jiwei-1013-01-464st-master-2
      INFO Deleted disk jiwei-1013-01-464st-master-2
      INFO Deleted disk jiwei-1013-01-464st-master-1
      INFO Deleted disk jiwei-1013-01-464st-worker-b-pmg5z
      INFO Deleted disk jiwei-1013-01-464st-master-0
      INFO Deleted disk jiwei-1013-01-464st-worker-a-csg2j
      INFO Deleted address jiwei-1013-01-464st-cluster-public-ip
      INFO Deleted address jiwei-1013-01-464st-cluster-ip
      INFO Deleted forwarding rule a516d89f9a4f14bdfb55a525b1a12a91
      INFO Deleted forwarding rule jiwei-1013-01-464st-api
      INFO Deleted forwarding rule jiwei-1013-01-464st-api-internal
      INFO Deleted target pool a516d89f9a4f14bdfb55a525b1a12a91
      INFO Deleted target pool jiwei-1013-01-464st-api
      INFO Deleted backend service jiwei-1013-01-464st-api-internal
      INFO Deleted instance group jiwei-1013-01-464st-master-us-central1-a
      INFO Deleted instance group jiwei-1013-01-464st-master-us-central1-c
      INFO Deleted instance group jiwei-1013-01-464st-master-us-central1-b
      INFO Deleted health check jiwei-1013-01-464st-api-internal
      INFO Deleted HTTP health check a516d89f9a4f14bdfb55a525b1a12a91
      INFO Deleted HTTP health check jiwei-1013-01-464st-api
      INFO Time elapsed: 4m18s   
      $ 
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules list --filter='network=installer-shared-vpc'
      NAME                                          NETWORK               DIRECTION  PRIORITY  ALLOW                                                                                                                                                     DENY  DISABLED
      ci-op-xpn-ingress-common                      installer-shared-vpc  INGRESS    60000     tcp:6443,tcp:22,tcp:80,tcp:443,icmp                                                                                                                             False
      ci-op-xpn-ingress-health-checks               installer-shared-vpc  INGRESS    60000     tcp:30000-32767,udp:30000-32767,tcp:6080,tcp:6443,tcp:22624,tcp:32335                                                                                           False
      ci-op-xpn-ingress-internal-network            installer-shared-vpc  INGRESS    60000     udp:4789,udp:6081,udp:500,udp:4500,esp,tcp:9000-9999,udp:9000-9999,tcp:10250,tcp:30000-32767,udp:30000-32767,tcp:10257,tcp:10259,tcp:22623,tcp:2379-2380        False
      jiwei-1013-01-464st-api                       installer-shared-vpc  INGRESS    1000      tcp:6443                                                                                                                                                        False
      jiwei-1013-01-464st-control-plane             installer-shared-vpc  INGRESS    1000      tcp:22623,tcp:10257,tcp:10259                                                                                                                                   False
      jiwei-1013-01-464st-etcd                      installer-shared-vpc  INGRESS    1000      tcp:2379-2380                                                                                                                                                   False
      jiwei-1013-01-464st-health-checks             installer-shared-vpc  INGRESS    1000      tcp:6080,tcp:6443,tcp:22624                                                                                                                                     False
      jiwei-1013-01-464st-internal-cluster          installer-shared-vpc  INGRESS    1000      tcp:30000-32767,udp:9000-9999,udp:30000-32767,udp:4789,udp:6081,tcp:9000-9999,udp:500,udp:4500,esp,tcp:10250                                                    False
      jiwei-1013-01-464st-internal-network          installer-shared-vpc  INGRESS    1000      icmp,tcp:22                                                                                                                                                     False
      k8s-a516d89f9a4f14bdfb55a525b1a12a91-http-hc  installer-shared-vpc  INGRESS    1000      tcp:30268                                                                                                                                                       False
      k8s-fw-a516d89f9a4f14bdfb55a525b1a12a91       installer-shared-vpc  INGRESS    1000      tcp:80,tcp:443                                                                                                                                                  FalseTo show all fields of the firewall, please show in JSON format: --format=json
      To show all fields in table format, please see the examples in --help.
      $ 
      
      FYI manually deleting those firewall-rules in the shared VPC does work.
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-api
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-api].
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-control-plane
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-control-plane].
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-etcd
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-etcd].
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-health-checks
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-health-checks].
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-internal-cluster
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-internal-cluster].
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q jiwei-1013-01-464st-internal-network
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/jiwei-1013-01-464st-internal-network].
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q k8s-a516d89f9a4f14bdfb55a525b1a12a91-http-hc
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/k8s-a516d89f9a4f14bdfb55a525b1a12a91-http-hc].
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q k8s-fw-a516d89f9a4f14bdfb55a525b1a12a91
      Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-a516d89f9a4f14bdfb55a525b1a12a91].
      $ 
      $ gcloud --project openshift-qe-shared-vpc compute firewall-rules list --filter='network=installer-shared-vpc'
      NAME                                NETWORK               DIRECTION  PRIORITY  ALLOW                                                                                                                                                     DENY  DISABLED
      ci-op-xpn-ingress-common            installer-shared-vpc  INGRESS    60000     tcp:6443,tcp:22,tcp:80,tcp:443,icmp                                                                                                                             False
      ci-op-xpn-ingress-health-checks     installer-shared-vpc  INGRESS    60000     tcp:30000-32767,udp:30000-32767,tcp:6080,tcp:6443,tcp:22624,tcp:32335                                                                                           False
      ci-op-xpn-ingress-internal-network  installer-shared-vpc  INGRESS    60000     udp:4789,udp:6081,udp:500,udp:4500,esp,tcp:9000-9999,udp:9000-9999,tcp:10250,tcp:30000-32767,udp:30000-32767,tcp:10257,tcp:10259,tcp:22623,tcp:2379-2380        FalseTo show all fields of the firewall, please show in JSON format: --format=json
      To show all fields in table format, please see the examples in --help.
      $ 
      

       

       

       

       

      Attachments

        Activity

          People

            rh-ee-bbarbach Brent Barbachem
            rhn-support-jiwei Jianli Wei
            Jianli Wei Jianli Wei
            Darragh Fitzmaurice Darragh Fitzmaurice
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: