Uploaded image for project: 'openstack-k8s'
  1. openstack-k8s
  2. OSPK8-806

SNAT with VLAN Networks

XMLWordPrintable

    • False
    • None
    • False

      A customer is attempting to set up Shift-on-Stack (IPI) on a cloud provisioned by OSPDirO.

      During the bootstrap phase, three master instances are provisioned. These instances need to communicate northbound out of the tenant network, expecting to use SNAT via the Neutron router, which has an IP on the cloud's external network.

      By default, SNAT traffic is sent across the Geneve tunnel to the control plane, which holds the external gateway chassis attribute.

      The issue arises when compute nodes configure the external network in Neutron as a VLAN segment. Packets on this network expect 802.1Q encapsulation, which is typically handled correctly since compute nodes are configured with VLAN interfaces, ensuring proper encapsulation of traffic from instances. This setup works as expected, confirmed by assigning a floating IP to an instance with DVR deployed in the cloud, causing traffic to originate from the compute host instead of traversing the tunnel.

      However, with SNAT, traffic goes across the Geneve tunnel and attempts to exit to the external network from the control plane host. When deployed by OSPDirO, this host is a VM with interfaces part of a bridge that handles the encapsulation. Consequently, VLAN-encapsulated traffic is emitted from an interface on the control plane that is already tagged.

       

      The solution we implemented was to modify the OVNCMSOptions for the cloud and make SNAT traffic follow the same DVR datapath as floating IPs.  

      The following template was added as a stack customization:

      # controllerCustomization.yaml
      parameter_defaults:
        # We have updated roles_data.yaml with this parameter however
        # roles_data is evaluated earlier in the stack deploy and overwritten
        # by ovn-dvr and associated templates that has Controller set as
        # "enable-chassis-as-gw"
        #
        # By defining this here we guarantee to have the last word
        ControllerParameters:
          OVNCMSOPtions: "disable-chassis-as-gw" 

      We also modified the compute role to set OVNCMSOptions: "enable-chassis-as-gw"

      It probably is worth using this in general since it doesn't make sense to forward an entire AZs SNAT traffic through 3 VMs for performance reasons.

      Related bz: https://bugzilla.redhat.com/show_bug.cgi?id=2084363

      EDIT –

      Cloud networking (provider networks/neutron rbac/subnets) are provisioned as a policy stack from the admin tenant.

      In this sanitized template these networks created in admin are consumed by two tenants unix-env and windows env.

      The neutron network zzz-customer-provider-net-vlan-3501 was being double tagged when being forwarded north to the controller vms in openshift.

       

      heat_template_version: wallaby
      description: initialize vlan provider networks for zzz-customer
      
      parameters:
        unix-env:
          type: string
        windows-env:
          type: string
      
      resources:
      
        # Default floating ip / snat network
        zzz-customer-provider-net-vlan-3501:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-3501
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 3501
            shared: true
            port_security_enabled: true
            router_external: true
      
        # Manila NFS Direct Network
        zzz-customer-provider-net-vlan-3507:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-3507
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 3507
            shared: true
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-32:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-32
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 32
            shared: false
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-33:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-33
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 33
            shared: false
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-36:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-36
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 36
            shared: false
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-38:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-38
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 38
            shared: false
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-96:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-96
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 96
            shared: false
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-97:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-97
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 97
            shared: false
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-98:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-98
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 98
            shared: false
            port_security_enabled: true
            router_external: true
      
        zzz-customer-provider-net-vlan-148:
          type: OS::Neutron::ProviderNet
          properties:
            name: zzz-customer-vlan-148
            network_type: vlan
            physical_network: datacentre
            segmentation_id: 148
            shared: false
            port_security_enabled: true
            router_external: true
      
        # Enable OpenStack IPAM for the External Network
        zzz-customer-subnet-vlan-3501:
          type: OS::Neutron::Subnet
          properties:
            name: zzz-customer-vlan-3501-subnet
            network: { get_resource: zzz-customer-provider-net-vlan-3501 }
            cidr: 172.16.202.0/24
            ip_version: 4
            allocation_pools:
              - start: 172.16.202.100
                end: 172.16.202.250
            gateway_ip: 172.16.202.1
            enable_dhcp: true
            dns_nameservers:
              - 172.18.70.27
              - 172.17.70.27
      
      
        # Enable OpenStack IPAM for the Manila NFS Direct Network
        zzz-customer-subnet-vlan-3507:
          type: OS::Neutron::Subnet
          properties:
            name: zzz-customer-vlan-3507-subnet
            network: { get_resource: zzz-customer-provider-net-vlan-3507 }
            cidr: 172.16.203.192/26
            ip_version: 4
            allocation_pools:
              - start: 172.16.203.195
                end: 172.16.203.250
            # gateway_ip: 172.16.203.193
            enable_dhcp: true
      
        # Do not enable/use OpenStack IPAM in the lab  (for vm networks)
        zzz-customer-subnet-vlan-32:
          type: OS::Neutron::Subnet
          properties:
            network: { get_resource: zzz-customer-provider-net-vlan-32 }
            cidr: 172.16.32.0/24
            ip_version: 4
            allocation_pools: [ ]
            gateway_ip: 172.16.32.1
            enable_dhcp: false
      
        zzz-customer-subnet-vlan-33:
          type: OS::Neutron::Subnet
          properties:
            network: { get_resource: zzz-customer-provider-net-vlan-33 }
            cidr: 172.16.33.0/24
            ip_version: 4
            allocation_pools: [ ]
            gateway_ip: 172.16.33.1
            enable_dhcp: false
      
        zzz-customer-subnet-vlan-36:
          type: OS::Neutron::Subnet
          properties:
            network: { get_resource: zzz-customer-provider-net-vlan-36 }
            cidr: 172.16.36.0/24
            ip_version: 4
            allocation_pools: [ ]
            gateway_ip: 172.16.36.1
            enable_dhcp: false
      
        zzz-customer-subnet-vlan-38:
          type: OS::Neutron::Subnet
          properties:
            network: { get_resource: zzz-customer-provider-net-vlan-38 }
            cidr: 172.16.38.0/24
            ip_version: 4
            allocation_pools: [ ]
            gateway_ip: 172.16.38.1
            enable_dhcp: false
      
        zzz-customer-subnet-vlan-96:
           type: OS::Neutron::Subnet
           properties:
             network: { get_resource: zzz-customer-provider-net-vlan-96 }
             cidr: 172.16.96.0/24
             ip_version: 4
             allocation_pools: [ ]
             gateway_ip: 172.16.96.1
             enable_dhcp: false
      
        zzz-customer-subnet-vlan-97:
           type: OS::Neutron::Subnet
           properties:
             network: { get_resource: zzz-customer-provider-net-vlan-97 }
             cidr: 172.16.97.0/24
             ip_version: 4
             allocation_pools: [ ]
             gateway_ip: 172.16.97.1
             enable_dhcp: false
      
        zzz-customer-subnet-vlan-98:
           type: OS::Neutron::Subnet
           properties:
             network: { get_resource: zzz-customer-provider-net-vlan-98 }
             cidr: 172.16.98.0/24
             ip_version: 4
             allocation_pools: [ ]
             gateway_ip: 172.16.98.1
             enable_dhcp: false
      
        zzz-customer-subnet-vlan-148:
           type: OS::Neutron::Subnet
           properties:
             network: { get_resource: zzz-customer-provider-net-vlan-148 }
             cidr: 172.16.148.0/24
             ip_version: 4
             allocation_pools: [ ]
             gateway_ip: 172.16.148.1
             enable_dhcp: false
      
      ### RBAC for controlling what tenants have access to networks defined in the admin project
      
        zzz-customer-provider-net-vlan-32-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-32 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-32-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-32 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
        zzz-customer-provider-net-vlan-33-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-33 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-33-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-33 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
        zzz-customer-provider-net-vlan-36-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-36 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-36-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-36 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
        zzz-customer-provider-net-vlan-38-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-38 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-38-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-38 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
        zzz-customer-provider-net-vlan-96-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-96 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-96-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-96 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
        zzz-customer-provider-net-vlan-97-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-97 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-97-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-97 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
        zzz-customer-provider-net-vlan-98-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-98 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-98-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-98 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
        zzz-customer-provider-net-vlan-148-rbac-ust-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-148 }
            object_type: network
            target_tenant: { get_param: unix-env }
      
        zzz-customer-provider-net-vlan-148-rbac-wst-dev:
          type: OS::Neutron::RBACPolicy
          properties:
            action: access_as_shared
            object_id: { get_resource: zzz-customer-provider-net-vlan-148 }
            object_type: network
            target_tenant: { get_param: windows-env }
      
      

              Unassigned Unassigned
              gprocuni@redhat.com Gregory Procunier
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated: