Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-39577

OpenStack cluster loops deprovisioning network

XMLWordPrintable

    • None
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Installed OpenStack OCP cluster via hive using:

      Attempted to delete via same hive build. Note that hive vendors installer for destroy, so the deprovision code was initially running from v0.9.0-master.0.20240828080122-811a267a59b1 (approx current tip of 4.14).

      Recognizing that this may not be supported (destroyer is supposed to be backward compatible, but not necessarily forward), I redeployed hive to quay.io/app-sre/hive:95efcb7a0a, current master build, where hive is vendored at v0.9.0-master.0.20240613201043-5e36f8fb1dde (I think this is a late-4.16/early-4.17 commit). The problem persists.

      So it's possible attempting to destroy with an older version put us in a bad state that the latest code can't recover from.

      Version-Release number of selected component (if applicable):

      OCP 4.16.6

      How reproducible:

      I installed the exact same cluster (same name, floating IP, etc.) earlier using hive from https://github.com/openshift/hive/pull/2436 which vendors installer at the same commit as #2438, and it destroyed successfully – at least as far as I know. It's possible that destroy left something behind that's causing the current problem. I have not tried to reproduce beyond that.

      Steps to Reproduce:

      1. Install cluster using hive (see attached manifests)
      2. Destroy cluster using hive

      Actual results:

      Destroyer pod loops with:

      time="2024-09-04T14:49:31Z" level=debug msg="Deleting openstack subnets"
time="2024-09-04T14:49:36Z" level=debug msg="Deleting openstack trunks"
time="2024-09-04T14:49:38Z" level=debug msg="Deleting Subnet: \"24d06989-8e12-4402-be5e-2a6eb3c0df51\""
time="2024-09-04T14:49:38Z" level=debug msg="Deleting Subnet \"24d06989-8e12-4402-be5e-2a6eb3c0df51\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/24d06989-8e12-4402-be5e-2a6eb3c0df51], but got 409 instead\n{\"NeutronError\": {\"type\": \"SubnetInUse\", \"message\": \"Unable to complete operation on subnet 24d06989-8e12-4402-be5e-2a6eb3c0df51: One or more ports have an IP allocation from this subnet.\", \"detail\": \"\"}}"
time="2024-09-04T14:49:38Z" level=debug msg="Exiting deleting openstack subnets"
time="2024-09-04T14:49:44Z" level=debug msg="Deleting Trunk \"6feb86a2-ee08-4ab0-8d8d-b1a7f263c152\""
time="2024-09-04T14:49:44Z" level=debug msg="Deleting Trunk \"6feb86a2-ee08-4ab0-8d8d-b1a7f263c152\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/6feb86a2-ee08-4ab0-8d8d-b1a7f263c152], but got 409 instead\n{\"NeutronError\": {\"type\": \"TrunkInUse\", \"message\": \"Trunk 6feb86a2-ee08-4ab0-8d8d-b1a7f263c152 is currently in use.\", \"detail\": \"\"}}"
time="2024-09-04T14:49:44Z" level=debug msg="Deleting Trunk \"ad2bc6d9-0c47-43f3-b953-c22af28a131e\""
time="2024-09-04T14:49:45Z" level=debug msg="Deleting Trunk \"ad2bc6d9-0c47-43f3-b953-c22af28a131e\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/ad2bc6d9-0c47-43f3-b953-c22af28a131e], but got 409 instead\n{\"NeutronError\": {\"type\": \"TrunkInUse\", \"message\": \"Trunk ad2bc6d9-0c47-43f3-b953-c22af28a131e is currently in use.\", \"detail\": \"\"}}"
time="2024-09-04T14:49:45Z" level=debug msg="Exiting deleting openstack trunks"
time="2024-09-04T14:50:10Z" level=debug msg="Deleting openstack ports"
time="2024-09-04T14:50:19Z" level=debug msg="Deleting Port \"4d3352e8-ae9b-4d16-a787-7541fe4a9870\""
time="2024-09-04T14:50:19Z" level=debug msg="Deleting Port \"4d3352e8-ae9b-4d16-a787-7541fe4a9870\" failed with error: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/4d3352e8-ae9b-4d16-a787-7541fe4a9870], but got 409 instead\n{\"NeutronError\": {\"type\": \"PortInUseAsTrunkParent\", \"message\": \"Port 4d3352e8-ae9b-4d16-a787-7541fe4a9870 is currently a parent port for trunk 6feb86a2-ee08-4ab0-8d8d-b1a7f263c152.\", \"detail\": \"\"}}"
time="2024-09-04T14:50:19Z" level=debug msg="Deleting associated trunk"
time="2024-09-04T14:50:19Z" level=debug msg="Deleting Trunk \"6feb86a2-ee08-4ab0-8d8d-b1a7f263c152\""
time="2024-09-04T14:50:20Z" level=debug msg="Deleting Trunk \"6feb86a2-ee08-4ab0-8d8d-b1a7f263c152\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/6feb86a2-ee08-4ab0-8d8d-b1a7f263c152], but got 409 instead\n{\"NeutronError\": {\"type\": \"TrunkInUse\", \"message\": \"Trunk 6feb86a2-ee08-4ab0-8d8d-b1a7f263c152 is currently in use.\", \"detail\": \"\"}}"
time="2024-09-04T14:50:20Z" level=debug msg="Exiting deleting associated trunk"
time="2024-09-04T14:50:20Z" level=debug msg="Deleting Port \"fe171ee2-0064-4a8a-9bcc-2e9ff27dd007\""
time="2024-09-04T14:50:20Z" level=debug msg="Deleting Port \"fe171ee2-0064-4a8a-9bcc-2e9ff27dd007\" failed with error: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/fe171ee2-0064-4a8a-9bcc-2e9ff27dd007], but got 409 instead\n{\"NeutronError\": {\"type\": \"PortInUseAsTrunkParent\", \"message\": \"Port fe171ee2-0064-4a8a-9bcc-2e9ff27dd007 is currently a parent port for trunk ad2bc6d9-0c47-43f3-b953-c22af28a131e.\", \"detail\": \"\"}}"
time="2024-09-04T14:50:20Z" level=debug msg="Deleting associated trunk"
time="2024-09-04T14:50:20Z" level=debug msg="Deleting Trunk \"ad2bc6d9-0c47-43f3-b953-c22af28a131e\""
time="2024-09-04T14:50:20Z" level=debug msg="Deleting Trunk \"ad2bc6d9-0c47-43f3-b953-c22af28a131e\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/ad2bc6d9-0c47-43f3-b953-c22af28a131e], but got 409 instead\n{\"NeutronError\": {\"type\": \"TrunkInUse\", \"message\": \"Trunk ad2bc6d9-0c47-43f3-b953-c22af28a131e is currently in use.\", \"detail\": \"\"}}"
time="2024-09-04T14:50:20Z" level=debug msg="Exiting deleting associated trunk"
time="2024-09-04T14:50:20Z" level=debug msg="Exiting deleting openstack ports"
time="2024-09-04T14:53:31Z" level=debug msg="Deleting openstack networks"
time="2024-09-04T14:53:38Z" level=debug msg="Deleting network: \"84c03de2-25d5-4357-835a-f9f737c2bc3d\""
time="2024-09-04T14:53:38Z" level=debug msg="Deleting Network \"84c03de2-25d5-4357-835a-f9f737c2bc3d\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/84c03de2-25d5-4357-835a-f9f737c2bc3d], but got 409 instead\n{\"NeutronError\": {\"type\": \"NetworkInUse\", \"message\": \"Unable to complete operation on network 84c03de2-25d5-4357-835a-f9f737c2bc3d. There are one or more ports still in use on the network.\", \"detail\": \"\"}}"
time="2024-09-04T14:53:45Z" level=debug msg="Deleting openstack ports"
time="2024-09-04T14:53:59Z" level=debug msg="Deleting Port \"fe171ee2-0064-4a8a-9bcc-2e9ff27dd007\""
time="2024-09-04T14:54:00Z" level=debug msg="Deleting Port \"fe171ee2-0064-4a8a-9bcc-2e9ff27dd007\" failed with error: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/fe171ee2-0064-4a8a-9bcc-2e9ff27dd007], but got 409 instead\n{\"NeutronError\": {\"type\": \"PortInUseAsTrunkParent\", \"message\": \"Port fe171ee2-0064-4a8a-9bcc-2e9ff27dd007 is currently a parent port for trunk ad2bc6d9-0c47-43f3-b953-c22af28a131e.\", \"detail\": \"\"}}"
time="2024-09-04T14:54:00Z" level=debug msg="Deleting associated trunk"
time="2024-09-04T14:54:00Z" level=debug msg="Deleting Port \"4d3352e8-ae9b-4d16-a787-7541fe4a9870\""
time="2024-09-04T14:54:00Z" level=debug msg="Deleting Trunk \"ad2bc6d9-0c47-43f3-b953-c22af28a131e\""
time="2024-09-04T14:54:00Z" level=debug msg="Deleting Port \"4d3352e8-ae9b-4d16-a787-7541fe4a9870\" failed with error: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/4d3352e8-ae9b-4d16-a787-7541fe4a9870], but got 409 instead\n{\"NeutronError\": {\"type\": \"PortInUseAsTrunkParent\", \"message\": \"Port 4d3352e8-ae9b-4d16-a787-7541fe4a9870 is currently a parent port for trunk 6feb86a2-ee08-4ab0-8d8d-b1a7f263c152.\", \"detail\": \"\"}}"
time="2024-09-04T14:54:00Z" level=debug msg="Deleting associated trunk"
time="2024-09-04T14:54:00Z" level=debug msg="Deleting Trunk \"6feb86a2-ee08-4ab0-8d8d-b1a7f263c152\""
time="2024-09-04T14:54:00Z" level=debug msg="Deleting Trunk \"ad2bc6d9-0c47-43f3-b953-c22af28a131e\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/ad2bc6d9-0c47-43f3-b953-c22af28a131e], but got 409 instead\n{\"NeutronError\": {\"type\": \"TrunkInUse\", \"message\": \"Trunk ad2bc6d9-0c47-43f3-b953-c22af28a131e is currently in use.\", \"detail\": \"\"}}"
time="2024-09-04T14:54:00Z" level=debug msg="Exiting deleting associated trunk"
time="2024-09-04T14:54:00Z" level=debug msg="Deleting Trunk \"6feb86a2-ee08-4ab0-8d8d-b1a7f263c152\" failed: Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/6feb86a2-ee08-4ab0-8d8d-b1a7f263c152], but got 409 instead\n{\"NeutronError\": {\"type\": \"TrunkInUse\", \"message\": \"Trunk 6feb86a2-ee08-4ab0-8d8d-b1a7f263c152 is currently in use.\", \"detail\": \"\"}}"
time="2024-09-04T14:54:00Z" level=debug msg="Exiting deleting associated trunk"
time="2024-09-04T14:54:00Z" level=debug msg="Exiting deleting openstack ports"
time="2024-09-04T14:54:00Z" level=debug msg="Exiting deleting openstack networks"

      Expected results:

      Cluster gets destroyed (except for floating IP, which I expect to need to release).

        1. efried-rhos-clean.yaml
          3 kB
        2. clouds.yaml
          0.9 kB
        3. deprov.log
          132 kB

              maandre@redhat.com Martin André
              efried.openshift Eric Fried
              Itshak Brown Itshak Brown
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: