-
Bug
-
Resolution: Done-Errata
-
Critical
-
None
-
4.17.0
-
Important
-
None
-
Installer Sprint 257, Installer (PB) Sprint 258
-
2
-
Proposed
-
False
-
-
Release Note Not Required
-
In Progress
Description of problem:
Shared VPC installation using service account having all required permissions failed due to cluster operator ingress degraded, by telling error "error getting load balancer's firewall: googleapi: Error 403: Required 'compute.firewalls.get' permission for 'projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-a5b1f420669b3474d959cff80e8452dc'"
Version-Release number of selected component (if applicable):
4.17.0-0.nightly-multi-2024-08-07-221959
How reproducible:
Always
Steps to Reproduce:
1. "create install-config", then insert the interested settings (see [1]) 2. "create cluster" (see [2])
Actual results:
Installation failed, because cluster operator ingress degraded (see [2] and [3]). $ oc get co ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE ingress False True True 113m The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: LoadBalancerReady=False (SyncLoadBalancerFailed: The service-controller component is reporting SyncLoadBalancerFailed events like: Error syncing load balancer: failed to ensure load balancer: error getting load balancer's firewall: googleapi: Error 403: Required 'compute.firewalls.get' permission for 'projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-a5b1f420669b3474d959cff80e8452dc', forbidden... $ In fact the mentioned k8s firewall-rule doesn't exist in the host project (see [4]), and, the given service account does have enough permissions (see [6]).
Expected results:
Installation succeeds, and all cluster operators are healthy.
Additional info:
- blocks
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- is related to
-
-
- Closed
-
- relates to
-
CORS-3525 Provision GCP with CAPI (GA)
-
- Closed
-
- links to
-
RHEA-2024:6122
OpenShift Container Platform 4.18.z bug fix update
(1 links to)
[OCPBUGS-38152] [GCP CAPI install] Shared VPC installation using service account having all required permissions failed due to cluster operator ingress degraded
rhn-support-jiwei For the deletion issue you noticed, please make sure you are testing with the fix in https://github.com/openshift/installer/pull/8810. If its still a problem when using that fix it seems reasonable to create a bug for tracking.
bfournie@redhat.com Yes, I re-tested with 4.18.0-0.nightly-2024-08-19-002129, which does have https://github.com/openshift/installer/pull/8810, and filed a bug, see https://issues.redhat.com/browse/OCPBUGS-38689
The firewall destroy is also a permissions issue, but it pertains to permissions in the host project. I'm not sure how we've handled this in the past.
padillon Previously if the given service account doesn't have permissions provisioning firewall-rules in the host project, firewall-rules creation will be skipped during cluster installation. For example, see below 4.16 test.
Snippet of "ipi-install-install/build-log.txt"
2024-08-13 12:10:53+00:00 - Using the IAM service account of minimal permissions for deploying OCP cluster into GCP shared VPC... install-config.yaml ------------------- apiVersion: v1 metadata: name: ci-op-c8gjk70n-a66ca sshKey: | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX baseDomain: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX platform: gcp: projectID: XXXXXXXXXXXX region: us-central1 networkProjectID: XXXXXXXXXXXX-shared-vpc network: installer-shared-vpc controlPlaneSubnet: installer-shared-vpc-subnet-1 computeSubnet: installer-shared-vpc-subnet-2 controlPlane: architecture: arm64 name: master platform: gcp: type: t2a-standard-4 osDisk: diskType: pd-ssd diskSizeGB: 200 tags: - preserved-ipi-xpn-control-plane replicas: 3 compute: - architecture: arm64 name: worker replicas: 3 platform: gcp: type: t2a-standard-4 tags: - preserved-ipi-xpn-compute ...output omitted... level=info msg=Credentials loaded from environment variable "GOOGLE_CLOUD_KEYFILE_JSON", file "/tmp/secret/xpn_min_perm_passthrough.json" level=warning msg=failed to find permission compute.firewalls.create, skipping firewall rule creation ...output omitted...
The "ipi-deprovision-deprovision/build-log.txt", where we can see no log on trying to listing/deleting firewall-rules.
Deprovisioning cluster ... 2024-08-13 18:09:52+00:00 - Using the IAM service account of minimal permissions for deploying OCP cluster into GCP shared VPC... /tmp/secret/metadata.json Copying the installation artifacts to the Installer's asset directory... Running the Installer's 'destroy cluster' command... level=info msg=Credentials loaded from environment variable "GOOGLE_CLOUD_KEYFILE_JSON", file "/tmp/secret/xpn_min_perm_passthrough.json" level=info msg=Stopped instance ci-op-c8gjk70n-a66ca-p9msz-worker-b-tkz49 level=info msg=Stopped instance ci-op-c8gjk70n-a66ca-p9msz-worker-f-z6gcn level=info msg=Stopped instance ci-op-c8gjk70n-a66ca-p9msz-worker-a-lc4lr level=info msg=Stopped instance ci-op-c8gjk70n-a66ca-p9msz-master-2 level=info msg=Stopped instance ci-op-c8gjk70n-a66ca-p9msz-master-0 level=info msg=Stopped instance ci-op-c8gjk70n-a66ca-p9msz-master-1 level=info msg=Deleted disk ci-op-c8gjk70n-a66ca-p9msz-master-1 level=info msg=Deleted disk ci-op-c8gjk70n-a66ca-p9msz-worker-b-tkz49 level=info msg=Deleted IAM project role bindings level=info msg=Deleted service account projects/XXXXXXXXXXXX/serviceAccounts/ci-op-c8gjk70n-a66ca-p9msz-w@XXXXXXXXXXXX.iam.gserviceaccount.com level=info msg=Deleted 2 recordset(s) in zone qe level=info msg=Deleted 3 recordset(s) in zone ci-op-c8gjk70n-a66ca-p9msz-private-zone level=info msg=Deleted DNS zone ci-op-c8gjk70n-a66ca-p9msz-private-zone level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/93/932d0da437931580c9d7f118c259624e1c3ec8e81acb2acf5740d945e511ba78/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-8qmg8/mysql-29706/_layers/sha256/f579260de74cdd61666857bff68e870d8b7259f1812142bca15dc42ca433c72f/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-q86lf/test-49455/_manifests/revisions/sha256/b8a489ef48c1015fa8284b87f59f8ec857a42db775f122e7eff18275884b6c05/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/cli/_layers/sha256/f74b2e4e8c945c91ccd4c6f43477d53b1feef3f9a83c08e1af5b5bbf96580d94/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_manifests/revisions/sha256/ecc69b8e89c62b7c2068340e18fafa35a5800f938b448154b775d3118d9f4591/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/14/142793795cee36320687e187996b3a79316053c5fee7c7787e4434e78cebb845/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/4d/4db5df5162b66a4ed2fde6acd2de9133ce4d68f87683653931af46c8341f343f/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/dc/dc0338f5d29b0d160df8a085f6fb51c86052fd92e5202311500e226201d975ea/data level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/ruby/_layers/sha256/8857f4647e6affd0ff58d5af980c8e34d5b599ee6f0d427aad72bea614a6c9d7/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/e3/e397673675977260fdabf44166698bbeceb946410e072c081eb1f4f5fed18fac/data level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/cli/_manifests/revisions/sha256/50d6e8fbcd60060a5ad8b207166d568a84530a110011adfd74f604da358e0f31/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/e0/e0567b54f4e4275914d892f7156811009fed96bbb7f7d3aabd686058751fc2d4/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-c5s6t/myimage/_layers/sha256/f74b2e4e8c945c91ccd4c6f43477d53b1feef3f9a83c08e1af5b5bbf96580d94/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-fqghc/ruby-test12766-local/_manifests/revisions/sha256/f36ddae644e081a7c218c423c0b6cbad627947c66e6c04cf3c330e297fc4fffa/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/e94075cdb888c2f3b897117ade6b75229c03c2bd5935ef33e2cb1810e9207926/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-k5rsv/first/_manifests/revisions/sha256/bf920ca7f146b802e1c9a8aab1fba3a3fe601c56b075ecef90834c13b90bb5bb/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-krt7z/test-50925/_manifests/revisions/sha256/e1150a0ea575c730dc18133a62f3858d06f8856598451d9e25bbbbcfa437eec5/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-p82rj/mystream/_layers/sha256/537cfc3d77e9a28fe211464371bffe9ae0097e37979a15666fcab21eb125b558/link level=info msg=Deleted bucket object docker/registry/v2/repositories/imzec/python-sample/_layers/sha256/7a9958543371ea8b93d844b21ed98700f4bb42ba8d9ff42543234c74802c5714/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/d8/d8955d3e8e04900a695f484cedfd2fb469e4965201c5b6e7dc70afbe15a612e8/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/e1/e1150a0ea575c730dc18133a62f3858d06f8856598451d9e25bbbbcfa437eec5/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/ed/ed88bae007bd9c42610e465ab436d60b2457b10b735a1bc8e5daba38f548c026/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-c5s6t/myimage/_manifests/revisions/sha256/d0be16b555a3a821bca04e7953b4ecefdffa838676199c3f045bfef8b28f0f00/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-p82rj/mystream/_layers/sha256/321a09b060aafbd3920dd54cf470e60e599d2f3dadfaba3798e9144353bd0a73/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/2f/2f3c60305226f5b02f3a7830c5bc5e9973fec64692feba2d494d45a2f96d38a9/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-krt7z/test-50925/_layers/sha256/c2cf70554e629dd6e8f87b344b3797526b1ba7ed49fd40d82ef41a61fd4b2051/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_manifests/revisions/sha256/c26501b3c8c1d70f22c778cf643ee051a4e7d9b27063b77d184aabb70c413c29/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/88/8857f4647e6affd0ff58d5af980c8e34d5b599ee6f0d427aad72bea614a6c9d7/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/ca/ca1636478fe5b8e2a56600e24d6759147feb15020824334f4a798c1cb6ed58e2/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-b6s2x/deployment-local/_manifests/revisions/sha256/9d29ff0fdbbec33bb4eebb0dbe0d0f3860a856987e5481bb0fc39f3aba086184/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/b4/b4dc30b5ab98521ef354e8d3bdc2ca583d7507651dc0126d89ad545a9468bb67/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/ca/cae34425c559694a7a6f072c2486b4b9353f567f1591ab9460e507232f7eccce/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-c5s6t/myimage/_layers/sha256/0a1a725750067ef414061b6edd71fd9e9f2f0d6adbcdb3f2904d56797cdbe079/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/a2/a2c8ccf1dbe78623f6d831d843da81c6c0df9da6c6a1dc062b17850fd8ed2c03/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/d4b811080f31c7ef4de3b70ae0b58d2e426c5146b6194666368bf8dd831c774f/link level=info msg=Deleted bucket object docker/registry/v2/repositories/imzec/python-sample/_layers/sha256/39b8358648e5bedce52187a85e7b2c939b771a6efec95f523050e8188f64af3e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/l26kl/python-sample/_manifests/revisions/sha256/757d2cc5b81785a2b035dfb48971f40061f4811741743e73881218bf99cff077/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/5a/5ad2cdb533eff4e5524e7511ba737a12924abecc0d3a0401f342d870a762d72a/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/e24600bac48ce2157027e68df9e26a81754edc6f7e099c33c0a89de879dba5d9/link level=info msg=Deleted bucket object docker/registry/v2/repositories/l26kl/python-sample/_layers/sha256/581288a53a925cfa530da9f86862c08b72ceb983b5e232220e0b409004b7a7ae/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/25/25a105c046267354ca01223f3b8f81744dbdbf1fa53a221c4ad91d1279d6f9c2/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_manifests/revisions/sha256/943ce385ef2e6d56da9677a7930a0b1ca09339cb9a75cd91cfab81c3e547e0a8/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-b6s2x/deployment-local/_manifests/revisions/sha256/a49b661c1ca3b508c480f9ee65ea345545b7031f5e877f72c750d20fa051fb5e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/imzec/python-sample/_layers/sha256/376a7503a7e3f4f0174eab5f3cc8b99d957df631e9b39007ff8d3c38b15fb498/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_layers/sha256/88adad5ca0cd95535377abbc2d4e0ab0964a47b4dc7a6b56ba7b19997173c10b/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_manifests/revisions/sha256/a2c8ccf1dbe78623f6d831d843da81c6c0df9da6c6a1dc062b17850fd8ed2c03/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-4xz6x/test-41414/_layers/sha256/c2cf70554e629dd6e8f87b344b3797526b1ba7ed49fd40d82ef41a61fd4b2051/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_manifests/revisions/sha256/fa2eefb8e9266b7b29f3c1b7db43b4f725690dde8aa6db4ba1cb4ba8e8cf384e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/ruby/_layers/sha256/39b8358648e5bedce52187a85e7b2c939b771a6efec95f523050e8188f64af3e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_layers/sha256/22b065e45b2773ca291f50eb40cfd89cb93969b4cab7589f3ead365eb402f627/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/c2/c26501b3c8c1d70f22c778cf643ee051a4e7d9b27063b77d184aabb70c413c29/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/7a/7a9958543371ea8b93d844b21ed98700f4bb42ba8d9ff42543234c74802c5714/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_layers/sha256/18ea5ce2a404500d5dfadf661ec0566faf7380e56b05ed37735b93844ed5faaa/link level=info msg=Deleted bucket object docker/registry/v2/repositories/imzec/python-sample/_layers/sha256/d8955d3e8e04900a695f484cedfd2fb469e4965201c5b6e7dc70afbe15a612e8/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/31/312897d9e4ab0ba95c44b2ab937aaff59db4625f43e0747e0ae706449a1fd99e/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_layers/sha256/7b90bc9e0c05f4002206c3f18e14681982cf90ff35c65a4d19dbc6b4890032f1/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/ruby/_layers/sha256/376a7503a7e3f4f0174eab5f3cc8b99d957df631e9b39007ff8d3c38b15fb498/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-xktgn/mystream/_manifests/revisions/sha256/a44545b8ea041a86e76d1cfbc19c12f60fb650321d1f87da9a7ef559ed79ee6c/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/a4/a44545b8ea041a86e76d1cfbc19c12f60fb650321d1f87da9a7ef559ed79ee6c/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/feec96e115e3570a3306a7f9fe5bd8ad126e034793081ed6a8a7935f0a81da36/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-k5rsv/first/_layers/sha256/bffe63f0059eb501c7705ef086bb4fed44620c9a6ac80e1bf25a6d831c8a1cdb/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_layers/sha256/bf79158e1551cb3f6346f0983c65746a0769790f7b4192c4740e6026d024ba90/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/a3/a380b4bcbdb747f5477a8d26191371c8815714cf37e16091873c7077a57250ff/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/a2/a2cbc909eafade0b8877c2963f0f1ab2673d23237901ff01ded0f4872e88df9c/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/eb/eb5f928702517f4c74bda7489a8e6814da022892c90a0fd31256e0bd8d620f23/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/42/4200f438cf2e9446f6bcff9d67ceea1f69ed07a2f83363b7fb52529f7ddd8a83/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/ef/efc743359cb494e714946cbb3f74235ee3ff23a63e8df181a14c2b9e5d0c1ed6/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-c5s6t/myimage/_layers/sha256/312897d9e4ab0ba95c44b2ab937aaff59db4625f43e0747e0ae706449a1fd99e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-xktgn/mystream/_layers/sha256/8f792156b1762e689fc6089b256e841901e83d0709379cbb3ab94e475b27a797/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/ruby/_layers/sha256/d8955d3e8e04900a695f484cedfd2fb469e4965201c5b6e7dc70afbe15a612e8/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/e9/e94075cdb888c2f3b897117ade6b75229c03c2bd5935ef33e2cb1810e9207926/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/50/50d6e8fbcd60060a5ad8b207166d568a84530a110011adfd74f604da358e0f31/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/bf/bf112c240f24966420594c548dea9f09a863a4d0bdb310880b9aed7da339b3ca/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/f4ce92a382f8ee5355dddd6244487ef5246bdb325dc5dddd25d55afc14fc1a54/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_layers/sha256/552d1f2373af9bfe12033568ebbfb0ccbb0de11279f9a415a29207e264d7f4d9/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_layers/sha256/73fd78e44c68146e00b6f00b946d9e2d441faeded0ec8c6f4cc82d0c53393615/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_layers/sha256/cae34425c559694a7a6f072c2486b4b9353f567f1591ab9460e507232f7eccce/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/0a/0a1a725750067ef414061b6edd71fd9e9f2f0d6adbcdb3f2904d56797cdbe079/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-vtnnr/is24160-2-lookup/_manifests/revisions/sha256/b4dc30b5ab98521ef354e8d3bdc2ca583d7507651dc0126d89ad545a9468bb67/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/55/552d1f2373af9bfe12033568ebbfb0ccbb0de11279f9a415a29207e264d7f4d9/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/ad/ad05e436503eac5b53ad17d36d91e1e8504ce2c005c98b40db10f47ab4812b4d/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-p82rj/mystream/_layers/sha256/e0567b54f4e4275914d892f7156811009fed96bbb7f7d3aabd686058751fc2d4/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/22/22b065e45b2773ca291f50eb40cfd89cb93969b4cab7589f3ead365eb402f627/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/af/af52ec70236bbefd4d56545f0298b47e999b9b5d23fa2fcdb3ac646e0db167de/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/e2/e24600bac48ce2157027e68df9e26a81754edc6f7e099c33c0a89de879dba5d9/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-92whd/test24167/_layers/sha256/73fd78e44c68146e00b6f00b946d9e2d441faeded0ec8c6f4cc82d0c53393615/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-92whd/test24167/_manifests/revisions/sha256/a44545b8ea041a86e76d1cfbc19c12f60fb650321d1f87da9a7ef559ed79ee6c/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-fqghc/ruby-test12766-local/_layers/sha256/4db5df5162b66a4ed2fde6acd2de9133ce4d68f87683653931af46c8341f343f/link level=info msg=Deleted bucket object docker/registry/v2/repositories/imzec/python-sample/_layers/sha256/142793795cee36320687e187996b3a79316053c5fee7c7787e4434e78cebb845/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/e3/e3a832d4c5a37781108a8ab8cf126313e582f79549d2b6bc43d566a6ca6d118f/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/61/61f21454c7cc5f0b1df5ab74c8adc22ec8513a889a1a611fa8080d8388b830d1/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/b8/b8a489ef48c1015fa8284b87f59f8ec857a42db775f122e7eff18275884b6c05/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/dc35c765a7328bd99273ee471b6b108980755023d4bcb1927fa7717942a9d318/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-p82rj/mystream/_layers/sha256/552d1f2373af9bfe12033568ebbfb0ccbb0de11279f9a415a29207e264d7f4d9/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_layers/sha256/043068d544216c34d25e7372daaecbfb89f61787a0ceeed83449dc6846e5e979/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/9f/9fa3954bbddbbd6063154425abb837daa71377c7a47db5b0d8c628d04af9dbe9/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-xktgn/mystream/_layers/sha256/73fd78e44c68146e00b6f00b946d9e2d441faeded0ec8c6f4cc82d0c53393615/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/cli/_manifests/revisions/sha256/d0be16b555a3a821bca04e7953b4ecefdffa838676199c3f045bfef8b28f0f00/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/75/757d2cc5b81785a2b035dfb48971f40061f4811741743e73881218bf99cff077/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/d4/d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_layers/sha256/5bf5a7de0e88a2658c783e7c1df12e5ab8389cdad10e99d5defed606598e3bbb/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/09/09e5b7f1bf871e8188193794cafcea9300a63a61807c451951c5f8be95f63c98/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/94/943ce385ef2e6d56da9677a7930a0b1ca09339cb9a75cd91cfab81c3e547e0a8/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/a4/a49b661c1ca3b508c480f9ee65ea345545b7031f5e877f72c750d20fa051fb5e/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_layers/sha256/321a09b060aafbd3920dd54cf470e60e599d2f3dadfaba3798e9144353bd0a73/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/18/18ea5ce2a404500d5dfadf661ec0566faf7380e56b05ed37735b93844ed5faaa/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/d0/d0be16b555a3a821bca04e7953b4ecefdffa838676199c3f045bfef8b28f0f00/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/f7/f74b2e4e8c945c91ccd4c6f43477d53b1feef3f9a83c08e1af5b5bbf96580d94/data level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_layers/sha256/b03f9d54d66b17aab1068852e0b886328de145cda76d46eb1252d1484e2ed898/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/55/558e284f7a153aade0cab8380b61ab3d479333ebded6f9a83ba0f4db15eb08fc/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_layers/sha256/8f792156b1762e689fc6089b256e841901e83d0709379cbb3ab94e475b27a797/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/932d0da437931580c9d7f118c259624e1c3ec8e81acb2acf5740d945e511ba78/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-p82rj/mystream/_layers/sha256/5bf5a7de0e88a2658c783e7c1df12e5ab8389cdad10e99d5defed606598e3bbb/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/4d/4d32c9289beb1d0af7d2f3af5e6ce78bed35e64140355eb728a3c15ea62c6713/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_manifests/revisions/sha256/a44545b8ea041a86e76d1cfbc19c12f60fb650321d1f87da9a7ef559ed79ee6c/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/f5/f579260de74cdd61666857bff68e870d8b7259f1812142bca15dc42ca433c72f/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/da05700210f681c47e021b8f3ce45f244f696854cfd656f0cd46e59cb8ca9b7d/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_layers/sha256/e0567b54f4e4275914d892f7156811009fed96bbb7f7d3aabd686058751fc2d4/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-jwqgk/st/_manifests/revisions/sha256/4200f438cf2e9446f6bcff9d67ceea1f69ed07a2f83363b7fb52529f7ddd8a83/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_layers/sha256/a1acc2fd53cd4b8369b226d144c216d25d9c1a380745768f496afbc3f48a1298/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/d4/d4b811080f31c7ef4de3b70ae0b58d2e426c5146b6194666368bf8dd831c774f/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/7b/7b90bc9e0c05f4002206c3f18e14681982cf90ff35c65a4d19dbc6b4890032f1/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/c8/c863e4ff3f088b0f520617439c193d2cf9858a60691ad98f145dc2354c465aaa/data level=info msg=Deleted bucket object docker/registry/v2/repositories/l26kl/python-sample/_layers/sha256/7a9958543371ea8b93d844b21ed98700f4bb42ba8d9ff42543234c74802c5714/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/04/043068d544216c34d25e7372daaecbfb89f61787a0ceeed83449dc6846e5e979/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_layers/sha256/25a105c046267354ca01223f3b8f81744dbdbf1fa53a221c4ad91d1279d6f9c2/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/37/376a7503a7e3f4f0174eab5f3cc8b99d957df631e9b39007ff8d3c38b15fb498/data level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_manifests/revisions/sha256/09e5b7f1bf871e8188193794cafcea9300a63a61807c451951c5f8be95f63c98/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_manifests/revisions/sha256/5ad2cdb533eff4e5524e7511ba737a12924abecc0d3a0401f342d870a762d72a/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/tools/_manifests/revisions/sha256/a380b4bcbdb747f5477a8d26191371c8815714cf37e16091873c7077a57250ff/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/7734d2f7be989dfe94fb891565830b8101d8e9bdbcbce8e5a7e1b1932eb3ac3c/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/88/88adad5ca0cd95535377abbc2d4e0ab0964a47b4dc7a6b56ba7b19997173c10b/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/0a92fe4d3412152aa6b8e9d74163065fda41b4c77227f5f9c30b5fc8d0d16f26/link level=info msg=Deleted bucket object docker/registry/v2/repositories/l26kl/python-sample/_layers/sha256/39b8358648e5bedce52187a85e7b2c939b771a6efec95f523050e8188f64af3e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/cli/_layers/sha256/8deeda9a2ea315c5704ca2b803eb6706b28633e2ab99752eba7c731f49869e12/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/python/_manifests/revisions/sha256/2f3c60305226f5b02f3a7830c5bc5e9973fec64692feba2d494d45a2f96d38a9/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-92pdp/mystream3/_layers/sha256/bebfa9e8d9ade32eb0f325a8d30bb79ec50d6bfe9f305e696177a355d2f334ef/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-q86lf/test-49455/_layers/sha256/c2cf70554e629dd6e8f87b344b3797526b1ba7ed49fd40d82ef41a61fd4b2051/link level=info msg=Deleted bucket object docker/registry/v2/repositories/l26kl/python-sample/_layers/sha256/376a7503a7e3f4f0174eab5f3cc8b99d957df631e9b39007ff8d3c38b15fb498/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/f4/f4ce92a382f8ee5355dddd6244487ef5246bdb325dc5dddd25d55afc14fc1a54/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/32/321a09b060aafbd3920dd54cf470e60e599d2f3dadfaba3798e9144353bd0a73/data level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/cli/_layers/sha256/ca1636478fe5b8e2a56600e24d6759147feb15020824334f4a798c1cb6ed58e2/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/bf/bf79158e1551cb3f6346f0983c65746a0769790f7b4192c4740e6026d024ba90/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-c5s6t/myimage/_layers/sha256/ca1636478fe5b8e2a56600e24d6759147feb15020824334f4a798c1cb6ed58e2/link level=info msg=Deleted bucket object docker/registry/v2/repositories/l26kl/python-sample/_layers/sha256/d8955d3e8e04900a695f484cedfd2fb469e4965201c5b6e7dc70afbe15a612e8/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/ed88bae007bd9c42610e465ab436d60b2457b10b735a1bc8e5daba38f548c026/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/fa/fa2eefb8e9266b7b29f3c1b7db43b4f725690dde8aa6db4ba1cb4ba8e8cf384e/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/43/4383798025cde56ad0d232f6c97da3edea0074ad40ced804658fc847c5abcb3e/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/55/558a2eb041fad2627ab7106b69bb5e1bf30f66743cb8827bcf451aa16d51170a/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/be/bebfa9e8d9ade32eb0f325a8d30bb79ec50d6bfe9f305e696177a355d2f334ef/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-krt7z/test-50925/_layers/sha256/4d32c9289beb1d0af7d2f3af5e6ce78bed35e64140355eb728a3c15ea62c6713/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/0a/0a92fe4d3412152aa6b8e9d74163065fda41b4c77227f5f9c30b5fc8d0d16f26/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/bf112c240f24966420594c548dea9f09a863a4d0bdb310880b9aed7da339b3ca/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/53/537cfc3d77e9a28fe211464371bffe9ae0097e37979a15666fcab21eb125b558/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/5b/5bf5a7de0e88a2658c783e7c1df12e5ab8389cdad10e99d5defed606598e3bbb/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/77/7734d2f7be989dfe94fb891565830b8101d8e9bdbcbce8e5a7e1b1932eb3ac3c/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/c2/c2cf70554e629dd6e8f87b344b3797526b1ba7ed49fd40d82ef41a61fd4b2051/data level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/python/_layers/sha256/7a9958543371ea8b93d844b21ed98700f4bb42ba8d9ff42543234c74802c5714/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/ruby/_manifests/revisions/sha256/ad05e436503eac5b53ad17d36d91e1e8504ce2c005c98b40db10f47ab4812b4d/link level=info msg=Deleted bucket object docker/registry/v2/repositories/imzec/python-sample/_layers/sha256/efc743359cb494e714946cbb3f74235ee3ff23a63e8df181a14c2b9e5d0c1ed6/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/bf/bffe63f0059eb501c7705ef086bb4fed44620c9a6ac80e1bf25a6d831c8a1cdb/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-4xz6x/test-41414/_manifests/revisions/sha256/558e284f7a153aade0cab8380b61ab3d479333ebded6f9a83ba0f4db15eb08fc/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/python/_layers/sha256/c863e4ff3f088b0f520617439c193d2cf9858a60691ad98f145dc2354c465aaa/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/58/581288a53a925cfa530da9f86862c08b72ceb983b5e232220e0b409004b7a7ae/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/da/da05700210f681c47e021b8f3ce45f244f696854cfd656f0cd46e59cb8ca9b7d/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/f3/f36ddae644e081a7c218c423c0b6cbad627947c66e6c04cf3c330e297fc4fffa/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/4383798025cde56ad0d232f6c97da3edea0074ad40ced804658fc847c5abcb3e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-c5s6t/myimage/_layers/sha256/8deeda9a2ea315c5704ca2b803eb6706b28633e2ab99752eba7c731f49869e12/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/cli/_layers/sha256/0a1a725750067ef414061b6edd71fd9e9f2f0d6adbcdb3f2904d56797cdbe079/link level=info msg=Deleted bucket object docker/registry/v2/repositories/l26kl/python-sample/_layers/sha256/dc0338f5d29b0d160df8a085f6fb51c86052fd92e5202311500e226201d975ea/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-vtnnr/is24160-2-lookup/_layers/sha256/537cfc3d77e9a28fe211464371bffe9ae0097e37979a15666fcab21eb125b558/link level=info msg=Deleted bucket object docker/registry/v2/repositories/imzec/python-sample/_manifests/revisions/sha256/e397673675977260fdabf44166698bbeceb946410e072c081eb1f4f5fed18fac/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/9d/9d29ff0fdbbec33bb4eebb0dbe0d0f3860a856987e5481bb0fc39f3aba086184/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-47xmp/mystream/_layers/sha256/558a2eb041fad2627ab7106b69bb5e1bf30f66743cb8827bcf451aa16d51170a/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-p82rj/mystream/_manifests/revisions/sha256/b4dc30b5ab98521ef354e8d3bdc2ca583d7507651dc0126d89ad545a9468bb67/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-xktgn/mystream/_layers/sha256/bebfa9e8d9ade32eb0f325a8d30bb79ec50d6bfe9f305e696177a355d2f334ef/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/fe/feec96e115e3570a3306a7f9fe5bd8ad126e034793081ed6a8a7935f0a81da36/data level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/cli/_layers/sha256/312897d9e4ab0ba95c44b2ab937aaff59db4625f43e0747e0ae706449a1fd99e/link level=info msg=Deleted bucket object docker/registry/v2/repositories/openshift/ruby/_layers/sha256/a2cbc909eafade0b8877c2963f0f1ab2673d23237901ff01ded0f4872e88df9c/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/b0/b03f9d54d66b17aab1068852e0b886328de145cda76d46eb1252d1484e2ed898/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/bf/bf920ca7f146b802e1c9a8aab1fba3a3fe601c56b075ecef90834c13b90bb5bb/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-8qmg8/mysql-29706/_manifests/revisions/sha256/e3a832d4c5a37781108a8ab8cf126313e582f79549d2b6bc43d566a6ca6d118f/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/a1/a1acc2fd53cd4b8369b226d144c216d25d9c1a380745768f496afbc3f48a1298/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-8qmg8/mysql-29706/_layers/sha256/9fa3954bbddbbd6063154425abb837daa71377c7a47db5b0d8c628d04af9dbe9/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-q86lf/test-49455/_layers/sha256/af52ec70236bbefd4d56545f0298b47e999b9b5d23fa2fcdb3ac646e0db167de/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/8d/8deeda9a2ea315c5704ca2b803eb6706b28633e2ab99752eba7c731f49869e12/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/8f/8f792156b1762e689fc6089b256e841901e83d0709379cbb3ab94e475b27a797/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/e0/e0444ce29df7b754434fd4557ec395c5a1759e4af71c3615a81471e0c7edeadc/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-b6s2x/deployment-local/_layers/sha256/eb5f928702517f4c74bda7489a8e6814da022892c90a0fd31256e0bd8d620f23/link level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-jf5wn/ocp11138-test-vczbg/_layers/sha256/e0444ce29df7b754434fd4557ec395c5a1759e4af71c3615a81471e0c7edeadc/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/39/39b8358648e5bedce52187a85e7b2c939b771a6efec95f523050e8188f64af3e/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/73/73fd78e44c68146e00b6f00b946d9e2d441faeded0ec8c6f4cc82d0c53393615/data level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/dc/dc35c765a7328bd99273ee471b6b108980755023d4bcb1927fa7717942a9d318/data level=info msg=Deleted bucket object docker/registry/v2/repositories/e2e-test-default-image-registry-4xz6x/test-41414/_layers/sha256/61f21454c7cc5f0b1df5ab74c8adc22ec8513a889a1a611fa8080d8388b830d1/link level=info msg=Deleted bucket object docker/registry/v2/blobs/sha256/ec/ecc69b8e89c62b7c2068340e18fafa35a5800f938b448154b775d3118d9f4591/data level=info msg=Deleted bucket ci-op-c8gjk70n-a66ca-p9msz-image-registry-us-central1-jeucmotm level=info msg=Deleted address ci-op-c8gjk70n-a66ca-p9msz-cluster-public-ip level=info msg=Deleted instance ci-op-c8gjk70n-a66ca-p9msz-worker-f-z6gcn level=info msg=Deleted instance ci-op-c8gjk70n-a66ca-p9msz-master-0 level=info msg=Deleted instance ci-op-c8gjk70n-a66ca-p9msz-worker-a-lc4lr level=info msg=Deleted instance ci-op-c8gjk70n-a66ca-p9msz-master-1 level=info msg=Deleted instance ci-op-c8gjk70n-a66ca-p9msz-worker-b-tkz49 level=info msg=Deleted instance ci-op-c8gjk70n-a66ca-p9msz-master-2 level=info msg=Deleted disk pvc-11b70f55-71d6-4351-b195-2dee7d6c293b level=info msg=Deleted disk ci-op-c8gjk70n-a66ca-p9msz-master-0 level=info msg=Deleted disk ci-op-c8gjk70n-a66ca-p9msz-worker-a-lc4lr level=info msg=Deleted disk pvc-b6c11a3c-d973-47c8-a2da-564ffb6e4d7e level=info msg=Deleted disk pvc-ec92803c-0b22-428c-a2cc-6a93a65b8915 level=info msg=Deleted disk ci-op-c8gjk70n-a66ca-p9msz-master-2 level=info msg=Deleted disk ci-op-c8gjk70n-a66ca-p9msz-worker-f-z6gcn level=info msg=Deleted address ci-op-c8gjk70n-a66ca-p9msz-cluster-ip level=info msg=Deleted backend service ci-op-c8gjk70n-a66ca-p9msz-api-internal level=info msg=Deleted disk pvc-b5c1ea1c-bc37-4990-aebf-d6e95c5245e9 level=info msg=Deleted target pool a949f9bfc6f8d4862a6901a1a1b65f64 level=info msg=Deleted target pool ci-op-c8gjk70n-a66ca-p9msz-api level=info msg=Deleted instance group ci-op-c8gjk70n-a66ca-p9msz-master-us-central1-f level=info msg=Deleted instance group ci-op-c8gjk70n-a66ca-p9msz-master-us-central1-b level=info msg=Deleted instance group ci-op-c8gjk70n-a66ca-p9msz-master-us-central1-a level=info msg=Deleted health check ci-op-c8gjk70n-a66ca-p9msz-api-internal level=info msg=Deleted HTTP health check a949f9bfc6f8d4862a6901a1a1b65f64 level=info msg=Deleted HTTP health check ci-op-c8gjk70n-a66ca-p9msz-api level=info msg=Writing quota footprint to /tmp/installer/quota.json level=info msg=Time elapsed: 5m6s level=info msg=Uninstallation complete! Copying the Installer logs and metadata to the artifacts directory...
> Shall I file an OCP bug for "destroy cluster" issue, and file an OCP doc bug for the additional permissions needed? Please advise, thanks!
rhn-support-jiwei These both seem like permissions issues to me, so I think it could be a single bug. But creating separate bugs is fine as well.
Yes, please create separate bugs for these. The firewall destroy is also a permissions issue, but it pertains to permissions in the host project. I'm not sure how we've handled this in the past. We may want to ensure we don't fail the install when we don't have proper permissions to destroy in the host project.
rhn-support-jiwei For the deletion issue you noticed, please make sure you are testing with the fix in https://github.com/openshift/installer/pull/8810. If its still a problem when using that fix it seems reasonable to create a bug for tracking.
Verified with 4.18.0-0.nightly-2024-08-18-185553
(1) QE's Flexy-install job
08-19 09:50:54.911 [01:50:54] INFO> install-config.yaml:
08-19 09:50:54.911 ---
08-19 09:50:54.911 apiVersion: v1
08-19 09:50:54.911 controlPlane:
08-19 09:50:54.911 architecture: amd64
08-19 09:50:54.911 hyperthreading: Enabled
08-19 09:50:54.911 name: master
08-19 09:50:54.911 platform:
08-19 09:50:54.912 gcp:
08-19 09:50:54.912 osDisk:
08-19 09:50:54.912 diskType:
08-19 09:50:54.912 diskSizeGB:
08-19 09:50:54.912 replicas: 3
08-19 09:50:54.912 compute:
08-19 09:50:54.912 - architecture: amd64
08-19 09:50:54.912 hyperthreading: Enabled
08-19 09:50:54.912 name: worker
08-19 09:50:54.912 platform:
08-19 09:50:54.912 gcp:
08-19 09:50:54.912 osDisk:
08-19 09:50:54.912 diskType:
08-19 09:50:54.912 diskSizeGB:
08-19 09:50:54.912 replicas: 2
08-19 09:50:54.912 metadata:
08-19 09:50:54.912 name: jiwei-0819b
08-19 09:50:54.912 platform:
08-19 09:50:54.912 gcp:
08-19 09:50:54.912 region: us-central1
08-19 09:50:54.912 projectID: openshift-qe
08-19 09:50:54.912 networkProjectID: openshift-qe-shared-vpc
08-19 09:50:54.912 network: installer-shared-vpc
08-19 09:50:54.912 controlPlaneSubnet: installer-shared-vpc-subnet-1
08-19 09:50:54.912 computeSubnet: installer-shared-vpc-subnet-2
08-19 09:50:54.913 defaultMachinePlatform: {}
08-19 09:50:54.913 pullSecret: HIDDEN
08-19 09:50:54.913 networking:
08-19 09:50:54.913 clusterNetwork:
08-19 09:50:54.913 - cidr: 10.128.0.0/14
08-19 09:50:54.913 hostPrefix: 23
08-19 09:50:54.913 serviceNetwork:
08-19 09:50:54.913 - 172.30.0.0/16
08-19 09:50:54.913 machineNetwork:
08-19 09:50:54.913 - cidr: 10.0.0.0/16
08-19 09:50:54.913 publish: External
08-19 09:50:54.913 credentialsMode: Passthrough
08-19 09:50:54.913 baseDomain: qe.gcp.devcluster.openshift.com
08-19 09:50:54.913 sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKyo2CxXHRP3Q5Ay0ZOlxCNSuH3xCSB68exLwE9b1fbvnzHQLfczM2oMySmEKmAN/l+mDSbrXVqx5Aa+Q76nmPK31ALbwCw94dd6A5IeM6t9PguWiodosXXccm7CgAh61+CIM6FkbrSw8mEFlUd/5LqQoi5xe3Y4ioYinXgDRIcN2aNaKr/BDGyMsnn4l9w/gOf+pMRQdqOa/cctKEt7SzMtnONNYKTf9hV2XQegVYNFgbmVKJvog3BR9jm8pAlE8mcGtn1QNsnNcXVqXRDKj/Sx1B7YfS631PVUX6Wpt2r2nYBwvUprmvh2Iqs/qBpG38kKe4afXRG9RNX65/ETiS/EdFta+q96Sbk/GOUPcn+NbNVwDFTKBdP0c88oPtp13vF78Ggdprx+uoUj6NAhb/bsnm4B1uKv71c++e+QqfFfjTcmtaCoZDRBHNT0FW+B/HLBhBQAV0qseSnZ69HYHgup0aKAbIwmsW5yqFewdU0CIPfgbPM06lo3/YkIunDf2IeEjzjz8LOtNmj+qiEsOU4xbMhbt9aeE4e6W6sC2FQrgqq2KFI27IvOeJSq6JAoXEIl+A5ZEhmjIKBrgucCZeMINB2c07354jGocq1A5d/oMc3YDoCc+HqDOCvugfxi8gQh1rKSrhZkOsvQTSzaLqpjLZEQ3IQU5oeRrKHPfDEQ==
08-19 09:50:54.914 openshift-qe
...output omitted...
08-19 09:50:55.840 GCP_SERVICE_ACCOUNT=aos-qe-serviceaccount@openshift-qe.iam.gserviceaccount.com
08-19 09:50:55.841 PROJECT_NAME=openshift-qe
...output omitted...
08-19 10:30:46.373 NAME STATUS ROLES AGE VERSION
08-19 10:30:46.373 jiwei-0819b-95ppx-master-0 Ready control-plane,master 30m v1.30.3
08-19 10:30:46.373 jiwei-0819b-95ppx-master-1 Ready control-plane,master 30m v1.30.3
08-19 10:30:46.373 jiwei-0819b-95ppx-master-2 Ready control-plane,master 30m v1.30.3
08-19 10:30:46.373 jiwei-0819b-95ppx-worker-a-njnxb Ready worker 20m v1.30.3
08-19 10:30:46.374 jiwei-0819b-95ppx-worker-b-l2tlt Ready worker 20m v1.30.3
08-19 10:30:46.374 NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
08-19 10:30:46.374 version 4.18.0-0.nightly-2024-08-18-185553 True False 7m54s Cluster version is 4.18.0-0.nightly-2024-08-18-185553
08-19 10:30:46.374 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
08-19 10:30:46.374 authentication 4.18.0-0.nightly-2024-08-18-185553 True False False 8m18s
08-19 10:30:46.374 baremetal 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.374 cloud-controller-manager 4.18.0-0.nightly-2024-08-18-185553 True False False 30m
08-19 10:30:46.375 cloud-credential 4.18.0-0.nightly-2024-08-18-185553 True False False 31m
08-19 10:30:46.375 cluster-autoscaler 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.375 config-operator 4.18.0-0.nightly-2024-08-18-185553 True False False 27m
08-19 10:30:46.375 console 4.18.0-0.nightly-2024-08-18-185553 True False False 10m
08-19 10:30:46.375 control-plane-machine-set 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.375 csi-snapshot-controller 4.18.0-0.nightly-2024-08-18-185553 True False False 27m
08-19 10:30:46.375 dns 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.376 etcd 4.18.0-0.nightly-2024-08-18-185553 True False False 25m
08-19 10:30:46.376 image-registry 4.18.0-0.nightly-2024-08-18-185553 True False False 17m
08-19 10:30:46.376 ingress 4.18.0-0.nightly-2024-08-18-185553 True False False 16m
08-19 10:30:46.376 insights 4.18.0-0.nightly-2024-08-18-185553 True False False 21m
08-19 10:30:46.376 kube-apiserver 4.18.0-0.nightly-2024-08-18-185553 True False False 10m
08-19 10:30:46.376 kube-controller-manager 4.18.0-0.nightly-2024-08-18-185553 True False False 24m
08-19 10:30:46.376 kube-scheduler 4.18.0-0.nightly-2024-08-18-185553 True False False 23m
08-19 10:30:46.377 kube-storage-version-migrator 4.18.0-0.nightly-2024-08-18-185553 True False False 27m
08-19 10:30:46.377 machine-api 4.18.0-0.nightly-2024-08-18-185553 True False False 19m
08-19 10:30:46.377 machine-approver 4.18.0-0.nightly-2024-08-18-185553 True False False 27m
08-19 10:30:46.377 machine-config 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.377 marketplace 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.377 monitoring 4.18.0-0.nightly-2024-08-18-185553 True False False 9m52s
08-19 10:30:46.378 network 4.18.0-0.nightly-2024-08-18-185553 True False False 29m
08-19 10:30:46.378 node-tuning 4.18.0-0.nightly-2024-08-18-185553 True False False 19m
08-19 10:30:46.378 openshift-apiserver 4.18.0-0.nightly-2024-08-18-185553 True False False 14m
08-19 10:30:46.378 openshift-controller-manager 4.18.0-0.nightly-2024-08-18-185553 True False False 19m
08-19 10:30:46.378 openshift-samples 4.18.0-0.nightly-2024-08-18-185553 True False False 19m
08-19 10:30:46.378 operator-lifecycle-manager 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.379 operator-lifecycle-manager-catalog 4.18.0-0.nightly-2024-08-18-185553 True False False 26m
08-19 10:30:46.379 operator-lifecycle-manager-packageserver 4.18.0-0.nightly-2024-08-18-185553 True False False 19m
08-19 10:30:46.379 service-ca 4.18.0-0.nightly-2024-08-18-185553 True False False 27m
08-19 10:30:46.379 storage 4.18.0-0.nightly-2024-08-18-185553 True False False 27m
(2) QE's Flexy-destroy job
OpenShift Jira Bot
added a comment - Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
OpenShift Jira Bot
added a comment - Hi bfournie@redhat.com ,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified. padillon With more testing, it looks like the additional permissions required in the host project are:
- resourcemanager.projects.getIamPolicy
- resourcemanager.projects.setIamPolicy
See my installation below, where "create cluster" succeeds, unfortunately "destroy cluster" keeps telling warnings until I manually deleted the k8s firewall-rules in the host project. Shall I file an OCP bug for "destroy cluster" issue, and file an OCP doc bug for the additional permissions needed? Please advise, thanks!
(1) the service account given to the installer, see the permissions included in the custom role "projects/openshift-qe-shared-vpc/roles/resourcemanager.projects.get_set_IamPolicy"
$ list_roles.sh ipi-xpn-minpt-permissions@openshift-qe.iam.gserviceaccount.com openshift-qe-shared-vpc Running Command: gcloud projects get-iam-policy openshift-qe --flatten='bindings[].members' --format='table(bindings.role)' --filter='bindings.members:ipi-xpn-minpt-permissions@openshift-qe.iam.gserviceaccount.com' ROLE roles/compute.admin roles/compute.loadBalancerAdmin roles/dns.admin roles/file.editor roles/iam.roleViewer roles/iam.securityAdmin roles/iam.serviceAccountAdmin roles/iam.serviceAccountKeyAdmin roles/iam.serviceAccountUser roles/resourcemanager.tagAdmin roles/resourcemanager.tagUser roles/storage.admin Running Command: gcloud projects get-iam-policy openshift-qe-shared-vpc --flatten='bindings[].members' --format='table(bindings.role)' --filter='bindings.members:ipi-xpn-minpt-permissions@openshift-qe.iam.gserviceaccount.com' ROLE projects/openshift-qe-shared-vpc/roles/dns.networks.bindPrivateDNSZone projects/openshift-qe-shared-vpc/roles/resourcemanager.projects.get_set_IamPolicy roles/compute.networkUser $ gcloud iam roles describe --project openshift-qe-shared-vpc resourcemanager.projects.get_set_IamPolicy description: permissions of getting and settings IAM policies etag: BwYfyOkYCts= includedPermissions: - resourcemanager.projects.getIamPolicy - resourcemanager.projects.setIamPolicy name: projects/openshift-qe-shared-vpc/roles/resourcemanager.projects.get_set_IamPolicy stage: ALPHA title: resourcemanager.projects.get_set_IamPolicy $ gcloud iam roles describe --project openshift-qe-shared-vpc dns.networks.bindPrivateDNSZone etag: BwXq5Yskvx4= includedPermissions: - dns.networks.bindPrivateDNSZone name: projects/openshift-qe-shared-vpc/roles/dns.networks.bindPrivateDNSZone stage: ALPHA title: dns.networks.bindPrivateDNSZone $
(2) install-config snippet
$ ./openshift-install version
./openshift-install 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest
built from commit 75e765b7024037206367fa7eb6193948c573a0e9
release image registry.build05.ci.openshift.org/ci-ln-1nv1tvb/release@sha256:e112a99d147993099aa1494ee1d136b9f49c004c06ce3110f6fc97be7b4b8785
WARNING Release Image Architecture not detected. Release Image Architecture is unknown
release architecture unknown
default architecture amd64
$
$ gcloud config get account
ipi-xpn-minpt-permissions@openshift-qe.iam.gserviceaccount.com
$ gcloud config get project
openshift-qe
$
$ yq-3.3.0 r test4/install-config.yaml platform
gcp:
projectID: openshift-qe
region: us-central1
networkProjectID: openshift-qe-shared-vpc
network: installer-shared-vpc
controlPlaneSubnet: installer-shared-vpc-subnet-1
computeSubnet: installer-shared-vpc-subnet-2
$ yq-3.3.0 r test4/install-config.yaml credentialsMode
Passthrough
$ yq-3.3.0 r test4/install-config.yaml compute
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
gcp:
type: n2-standard-2
tags:
- preserved-ipi-xpn-compute
replicas: 2
$ yq-3.3.0 r test4/install-config.yaml controlPlane
architecture: amd64
hyperthreading: Enabled
name: master
platform:
gcp:
tags:
- preserved-ipi-xpn-control-plane
replicas: 3
$
(3) "create cluster"
$ ./openshift-install create cluster --dir test4 WARNING Release Image Architecture not detected. Release Image Architecture is unknown INFO Credentials loaded from file "~/.gcp/osServiceAccount.json" INFO Consuming Install Config from target directory WARNING failed to find permission compute.firewalls.create, skipping firewall rule creation INFO Adding clusters... INFO Creating infrastructure resources... INFO Started local control plane with envtest INFO Stored kubeconfig for envtest in: test4/.clusterapi_output/envtest.kubeconfig INFO setting "GOOGLE_APPLICATION_CREDENTIALS" to ~/.gcp/osServiceAccount.json for capg infrastructure controller INFO Running process: Cluster API with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:37863 --webhook-port=35107 --webhook-cert-dir=/tmp/envtest-serving-certs-1940422849 --kubeconfig=test4/.clusterapi_output/envtest.kubeconfig] INFO Running process: gcp infrastructure provider with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:44427 --webhook-port=33419 --webhook-cert-dir=/tmp/envtest-serving-certs-4074934004 --kubeconfig=test4/.clusterapi_output/envtest.kubeconfig] INFO Creating infra manifests... INFO Created manifest *v1.Namespace, namespace= name=openshift-cluster-api-guests INFO Created manifest *v1beta1.Cluster, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj INFO Created manifest *v1beta1.GCPCluster, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj INFO Done creating infra manifests INFO Creating kubeconfig entry for capi cluster jiwei-0816f-jppdj INFO Waiting up to 15m0s (until 5:13PM CST) for network infrastructure to become ready... INFO Network infrastructure is ready WARNING failed to find permission compute.firewalls.create, skipping firewall rule creation WARNING failed to find permission compute.firewalls.create, skipping firewall rule creation INFO Created manifest *v1beta1.GCPMachine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-bootstrap INFO Created manifest *v1beta1.GCPMachine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-master-0 INFO Created manifest *v1beta1.GCPMachine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-master-1 INFO Created manifest *v1beta1.GCPMachine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-master-2 INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-bootstrap INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-master-0 INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-master-1 INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-master-2 INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-bootstrap INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=jiwei-0816f-jppdj-master INFO Waiting up to 15m0s (until 5:15PM CST) for machines [jiwei-0816f-jppdj-bootstrap jiwei-0816f-jppdj-master-0 jiwei-0816f-jppdj-master-1 jiwei-0816f-jppdj-master-2] to provision... INFO Control-plane machines are ready INFO Cluster API resources have been created. Waiting for cluster to become ready... INFO Waiting up to 20m0s (until 5:21PM CST) for the Kubernetes API at https://api.jiwei-0816f.qe.gcp.devcluster.openshift.com:6443... INFO API v1.30.3 up INFO Waiting up to 45m0s (until 5:49PM CST) for bootstrapping to complete... INFO Destroying the bootstrap resources... WARNING Destroying GCP Bootstrap Resources WARNING failed to find permission compute.firewalls.create, skipping firewall rule creation INFO Waiting up to 5m0s for bootstrap machine deletion openshift-cluster-api-guests/jiwei-0816f-jppdj-bootstrap... INFO Shutting down local Cluster API controllers... INFO Stopped controller: Cluster API WARNING process cluster-api-provider-gcp exited with error: signal: killed INFO Stopped controller: gcp infrastructure provider INFO Shutting down local Cluster API control plane... INFO Local Cluster API system has completed operations INFO no post-destroy requirements for the gcp provider INFO Finished destroying bootstrap resources INFO Waiting up to 40m0s (until 6:00PM CST) for the cluster at https://api.jiwei-0816f.qe.gcp.devcluster.openshift.com:6443 to initialize... INFO Waiting up to 30m0s (until 5:59PM CST) to ensure each cluster operator has finished progressing... INFO All cluster operators have completed progressing INFO Checking to see if there is a route at openshift-console/console... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=test4/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.jiwei-0816f.qe.gcp.devcluster.openshift.com INFO Login to the console with user: "kubeadmin", and password: "<PASSWORD>" INFO Time elapsed: 35m0s $ export KUBECONFIG=test4/auth/kubeconfig $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False 72s Cluster version is 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest $ oc get nodes NAME STATUS ROLES AGE VERSION jiwei-0816f-jppdj-master-0 Ready control-plane,master 24m v1.30.3 jiwei-0816f-jppdj-master-1 Ready control-plane,master 24m v1.30.3 jiwei-0816f-jppdj-master-2 Ready control-plane,master 25m v1.30.3 jiwei-0816f-jppdj-worker-a-tpb5g Ready worker 11m v1.30.3 jiwei-0816f-jppdj-worker-b-j2pc2 Ready worker 11m v1.30.3 $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 2m35s baremetal 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m cloud-controller-manager 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 24m cloud-credential 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 27m cluster-autoscaler 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m config-operator 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 22m console 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 6m55s control-plane-machine-set 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 20m csi-snapshot-controller 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 22m dns 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m etcd 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 20m image-registry 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 10m ingress 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 9m55s insights 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 15m kube-apiserver 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 11m kube-controller-manager 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 18m kube-scheduler 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 17m kube-storage-version-migrator 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 22m machine-api 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 11m machine-approver 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m machine-config 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 20m marketplace 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m monitoring 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 3m46s network 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 23m node-tuning 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 11m openshift-apiserver 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 11m openshift-controller-manager 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 15m openshift-samples 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 10m operator-lifecycle-manager 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m operator-lifecycle-manager-catalog 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m operator-lifecycle-manager-packageserver 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 16m service-ca 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 22m storage 4.18.0-0.test-2024-08-16-041957-ci-ln-1nv1tvb-latest True False False 21m $
(4) "destroy cluster", note that it'll keep telling the warning, if I didn't delete the k8s firewall-rules manually from the host project (see "(5)")
$ ./openshift-install destroy cluster --dir test4 INFO Credentials loaded from file "~/.gcp/osServiceAccount.json" INFO Stopped instance jiwei-0816f-jppdj-worker-b-j2pc2 INFO Stopped instance jiwei-0816f-jppdj-worker-a-tpb5g INFO Stopped instance jiwei-0816f-jppdj-master-1 INFO Stopped instance jiwei-0816f-jppdj-master-2 INFO Stopped instance jiwei-0816f-jppdj-master-0 INFO Deleted IAM project role bindings INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0816f-jppdj-w@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0816f-jppdj-m@openshift-qe.iam.gserviceaccount.com INFO Deleted 2 recordset(s) in zone qe INFO Deleted 3 recordset(s) in zone jiwei-0816f-jppdj-private-zone INFO Deleted DNS zone jiwei-0816f-jppdj-private-zone INFO Deleted bucket jiwei-0816f-jppdj-image-registry-us-central1-rcgosetbupujwpeio INFO Deleted instance jiwei-0816f-jppdj-master-1 INFO Deleted instance jiwei-0816f-jppdj-worker-b-j2pc2 INFO Deleted instance jiwei-0816f-jppdj-master-2 INFO Deleted instance jiwei-0816f-jppdj-master-0 INFO Deleted instance jiwei-0816f-jppdj-worker-a-tpb5g INFO Deleted disk jiwei-0816f-jppdj-master-1 INFO Deleted disk jiwei-0816f-jppdj-worker-b-j2pc2 INFO Deleted disk jiwei-0816f-jppdj-master-2 INFO Deleted disk jiwei-0816f-jppdj-master-0 INFO Deleted disk jiwei-0816f-jppdj-worker-a-tpb5g INFO Deleted address jiwei-0816f-jppdj-api-internal INFO Deleted backend service jiwei-0816f-jppdj-api-internal INFO Deleted target pool aca054616921a4b7ab893220bc233b2c INFO Deleted target tcp proxy jiwei-0816f-jppdj-apiserver INFO Deleted HTTP health check aca054616921a4b7ab893220bc233b2c INFO Deleted backend service jiwei-0816f-jppdj-apiserver INFO Deleted instance group jiwei-0816f-jppdj-master-us-central1-b INFO Deleted instance group jiwei-0816f-jppdj-master-us-central1-f INFO Deleted instance group jiwei-0816f-jppdj-master-us-central1-c INFO Deleted instance group jiwei-0816f-jppdj-master-us-central1-a INFO Deleted health check jiwei-0816f-jppdj-apiserver INFO Deleted health check jiwei-0816f-jppdj-api-internal WARNING failed to delete firewall k8s-aca054616921a4b7ab893220bc233b2c-http-hc: googleapi: Error 403: Required 'compute.firewalls.delete' permission for 'projects/openshift-qe-shared-vpc/global/firewalls/k8s-aca054616921a4b7ab893220bc233b2c-http-hc' WARNING More details: WARNING Reason: forbidden, Message: Required 'compute.firewalls.delete' permission for 'projects/openshift-qe-shared-vpc/global/firewalls/k8s-aca054616921a4b7ab893220bc233b2c-http-hc' WARNING Reason: forbidden, Message: Required 'compute.networks.updatePolicy' permission for 'projects/openshift-qe-shared-vpc/global/networks/installer-shared-vpc' WARNING WARNING failed to delete firewall k8s-fw-aca054616921a4b7ab893220bc233b2c: googleapi: Error 403: Required 'compute.firewalls.delete' permission for 'projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-aca054616921a4b7ab893220bc233b2c' WARNING More details: WARNING Reason: forbidden, Message: Required 'compute.firewalls.delete' permission for 'projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-aca054616921a4b7ab893220bc233b2c' WARNING Reason: forbidden, Message: Required 'compute.networks.updatePolicy' permission for 'projects/openshift-qe-shared-vpc/global/networks/installer-shared-vpc' WARNING INFO Deleted firewall rule k8s-fw-aca054616921a4b7ab893220bc233b2c INFO Deleted firewall rule k8s-aca054616921a4b7ab893220bc233b2c-http-hc INFO Time elapsed: 12m2s INFO Uninstallation complete! $
(5) manually delete the k8s firewall-rules in the host project, so that "destroy cluster" could move on
$ gcloud --project openshift-qe-shared-vpc compute firewall-rules delete -q k8s-fw-aca054616921a4b7ab893220bc233b2c k8s-aca054616921a4b7ab893220bc233b2c-http-hc Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-aca054616921a4b7ab893220bc233b2c]. Deleted [https://www.googleapis.com/compute/v1/projects/openshift-qe-shared-vpc/global/firewalls/k8s-aca054616921a4b7ab893220bc233b2c-http-hc]. $
rhn-support-jiwei I was rereading my comments above and I think they are difficult to understand. To unblock your testing, I believe you need to add the Service Account Admin role in the network host project to the installer credentials (service account).
That should allow you to proceed. My comments are an initial attempt to summarize things for documentation, but I will need to edit/revise to make it easier to understand.
Correct, we do not consider tags when determining whether to create firewall rules. The creation of firewall rules is based entirely on the presence of permissions, in the control plane service account for ingress firewall rules and in the installer creds for the ssh firewall rules.
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Important: OpenShift Container Platform 4.18.1 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:6122