Loading...

XML

Word

Printable

Details

Type: Ticket
Resolution: Done
Priority: Normal
Fix Version/s: None
Affects Version/s: None
Component/s: Maistra
Labels:

Blocked:
False
Blocked Reason:
None
Ready:
False

SFDC Cases Links:
SFDC Cases Counter:

Products:

Red Hat OpenShift Service on Amazon

Description

Issue: Unable to create a Second AWS Application Load Balancer using the AWS Load Balancer Operator for OSSM (OpenShift Service Mesh) ingress gateway

**Customer is trying to route the traffic for one particular application through the second ALB.

Below is the existing flow for all apps running in OpenShift Service Mesh

All apps (*.wildcard.com) --> AWS ALB --> istio-default-ingress-gateway --> virtual service --> service --> pod (hosted in OSSM)

They are trying to add a new Second ALB along with above default Service Mesh Ingress Gateway but encountered an error. See below steps taken by the customer in arriving on the error.

sample.app.com (custom domain) --> AWS ALB --> istio-cloud-flare-ingress-gateway --> virtual service --> service --> pod (hosted in OSSM)

STEPS TAKEN BY THE CUSTOMER BUT ENCOUNTERED AN ISSUE AFTER.

Rolled out a second gateway via the Service Mesh control plane as stated below,

additionalIngress:
   cloud-flare-ingress-gateway:
     enabled: true
     runtime:
       container:
         resources:
           requests:
             cpu: 10m
             memory: 300Mi
           limits:
             cpu: '2'
             memory: 1Gi
       deployment:
         autoScaling:
           maxReplicas: 4
           minReplicas: 2
           targetCPUUtilizationPercentage: 80
           enabled: true
     service:
       metadata:
         labels:
           app: cloud-flare-ingress-gateway
           istio: cloud-flare-ingress-gateway
       ports:
         - name: status-port
           port: 15020
           targetPort: 0
         - name: https
           port: 443
           targetPort: 8443
       type: NodePort

2. Created a cloud-flare-ingress-gateway as below,

kind: Gateway
apiVersion: networking.istio.io/v1beta1
metadata:
  name: cloud-flare-gateway
  namespace: isito-system
spec:
  servers:
    - port:
        number: 443
        protocol: HTTPS
        name: https
      hosts:
        - '*'
      tls:
        mode: SIMPLE
        credentialName: istio-ingress-gateway-cert
    - port:
        number: 80
        protocol: HTTP
        name: http
      hosts:
        - '*'
  selector:
    istio: cloud-flare-ingress-gateway

3. Tried creating the second AWS ALB via an ingress file (CLOUD-FLARE-ISTIO-INGRESS-ALB.yaml),

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  annotations:
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/success-codes: '200'
    alb.ingress.kubernetes.io/load-balancer-name: lilly-kubed-dev-cloud-flare-alb
    alb.ingress.kubernetes.io/backend-protocol-version: HTTP1
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/shield-advanced-protection: 'true'
    alb.ingress.kubernetes.io/healthcheck-port: '31855'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: >-
      arn:aws:acm:us-east-2:xxxxxxxxxxxxxx:certificate/109580a4-d029-4152-9a37-d442cb3114c6
    alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
    alb.ingress.kubernetes.io/inbound-cidrs: '0.0.0.0/0, 10.2.0.0/16'
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
  name: alb-ingress-cloud-flare-gw
  namespace: istio-system
  labels:
    app: lilly-kubed-dev-external-alb-cloud-flare-gw
    app.kubernetes.io/instance: service-mesh
spec:
  ingressClassName: cloud-flare
  tls:
    - hosts:
        - lillydirect.cloudflare-poc.apps.lilly.com
      secretName: istio-ingress-gateway-cert
  rules:
    - host: lillydirect.cloudflare-poc.apps.lilly.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: cloud-flare-ingress-gateway
                port:
                  number: 443

The AWS LB controller is creating and deleting the second ALB by itself as per the below logs.

{"level":"info","ts":1711008032.6249585,"logger":"controllers.ingress","msg":"creating securityGroup","resourceID":"ManagedLBSecurityGroup"}
{"level":"info","ts":1711008032.8492339,"logger":"controllers.ingress","msg":"created securityGroup","resourceID":"ManagedLBSecurityGroup","securityGroupID":"sg-094c1bbef9bbec891"} {"level":"info","ts":1711008032.9297352,"msg":"authorizing securityGroup ingress","securityGroupID":"sg-094c1bbef9bbec891","permission":[{"FromPort":443,"IpProtocol":"tcp","IpRanges":[
{"CidrIp":"0.0.0.0/0","Description":""}
],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":443,"UserIdGroupPairs":null},{"FromPort":443,"IpProtocol":"tcp","IpRanges":[
{"CidrIp":"10.2.0.0/16","Description":""}
],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":443,"UserIdGroupPairs":null},{"FromPort":80,"IpProtocol":"tcp","IpRanges":[
{"CidrIp":"0.0.0.0/0","Description":""}
],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":80,"UserIdGroupPairs":null},{"FromPort":80,"IpProtocol":"tcp","IpRanges":[
{"CidrIp":"10.2.0.0/16","Description":""}
],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":80,"UserIdGroupPairs":null}]} {"level":"info","ts":1711008033.099912,"msg":"authorized securityGroup ingress","securityGroupID":"sg-094c1bbef9bbec891"} {"level":"info","ts":1711008033.244688,"logger":"controllers.ingress","msg":"creating targetGroup","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"istio-system/alb-ingress-cloud-flare-gw-cloud-flare-ingress-gateway:443"} {"level":"info","ts":1711008033.4823337,"logger":"controllers.ingress","msg":"created targetGroup","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"istio-system/alb-ingress-cloud-flare-gw-cloud-flare-ingress-gateway:443","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:targetgroup/k8s-istiosys-cloudfla-77001729f9/15d7bc13ffdb0974"} {"level":"info","ts":1711008033.6756036,"logger":"controllers.ingress","msg":"creating loadBalancer","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"LoadBalancer"} {"level":"info","ts":1711008034.416678,"logger":"controllers.ingress","msg":"created loadBalancer","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:loadbalancer/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391"} {"level":"info","ts":1711008034.4976041,"logger":"controllers.ingress","msg":"creating listener","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"80"} {"level":"info","ts":1711008034.596129,"logger":"controllers.ingress","msg":"created listener","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"80","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:listener/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391/a3d95ce8a31f24ed"} {"level":"info","ts":1711008034.5961719,"logger":"controllers.ingress","msg":"creating listener","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"443"} {"level":"info","ts":1711008034.8539655,"logger":"controllers.ingress","msg":"created listener","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"443","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:listener/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391/0cd0341bc6ebe01e"} {"level":"info","ts":1711008035.020052,"logger":"controllers.ingress","msg":"creating listener rule","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"443:1"} {"level":"info","ts":1711008035.1240208,"logger":"controllers.ingress","msg":"created listener rule","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"443:1","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:listener-rule/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391/0cd0341bc6ebe01e/f39c69d0bfbfae1f"} {"level":"info","ts":1711008035.1240907,"logger":"controllers.ingress","msg":"creating targetGroupBinding","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"istio-system/alb-ingress-cloud-flare-gw-cloud-flare-ingress-gateway:443"} {"level":"info","ts":1711008035.1920443,"logger":"controllers.ingress","msg":"created targetGroupBinding","stackID":"istio-system/alb-ingress-cloud-flare-gw","resourceID":"istio-system/alb-ingress-cloud-flare-gw-cloud-flare-ingress-gateway:443","targetGroupBinding":{"namespace":"istio-system","name":"k8s-istiosys-cloudfla-77001729f9"}} {"level":"info","ts":1711008035.2850106,"logger":"controllers.ingress","msg":"enabling shield protection","resourceARN":"arn:aws:elasticloadbalancing:us-east-2:566053369253:loadbalancer/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391","protectionName":"managed by aws-load-balancer-controller"} {"level":"info","ts":1711008035.4763072,"logger":"controllers.ingress","msg":"enabled shield protection","resourceARN":"arn:aws:elasticloadbalancing:us-east-2:566053369253:loadbalancer/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391","protectionName":"managed by aws-load-balancer-controller","protectionID":"8234f694-8bd5-4f28-8723-d71dcc34db14"} {"level":"info","ts":1711008035.4763424,"logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"istio-system/alb-ingress-cloud-flare-gw"} {"level":"info","ts":1711008035.7271805,"msg":"registering targets","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:targetgroup/k8s-istiosys-cloudfla-77001729f9/15d7bc13ffdb0974","targets":[
{"AvailabilityZone":null,"Id":"i-00b97651353da83db","Port":31657}
,{"AvailabilityZone":null,"Id":"i-039f99869440eb9ed","Port":31657},{"AvailabilityZone":null,"Id":"i-03f7327fc33f34d70","Port":31657},{"AvailabilityZone":null,"Id":"i-049bdb2362160b076","Port":31657},{"AvailabilityZone":null,"Id":"i-0705e01abc70d1d7d","Port":31657},{"AvailabilityZone":null,"Id":"i-07238bec414c06d56","Port":31657},{"AvailabilityZone":null,"Id":"i-07758d9809dc98a3a","Port":31657},{"AvailabilityZone":null,"Id":"i-07c9540ef8de2d9c1","Port":31657},{"AvailabilityZone":null,"Id":"i-083336dfdb9d5db74","Port":31657},{"AvailabilityZone":null,"Id":"i-0889dc936ea100b3b","Port":31657},{"AvailabilityZone":null,"Id":"i-0cc122fe832e4a54f","Port":31657},{"AvailabilityZone":null,"Id":"i-0f3e1de4047f936bd","Port":31657}]} {"level":"info","ts":1711008035.925043,"msg":"registered targets","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:targetgroup/k8s-istiosys-cloudfla-77001729f9/15d7bc13ffdb0974"} {"level":"info","ts":1711008037.159926,"logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"istio-system/alb-ingress-cloud-flare-gw\",\"resources\":{}}"} {"level":"info","ts":1711008037.746406,"logger":"controllers.ingress","msg":"deleting loadBalancer","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:loadbalancer/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391"} {"level":"info","ts":1711008037.8446121,"logger":"controllers.ingress","msg":"deleted loadBalancer","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:loadbalancer/app/lilly-kubed-dev-cloud-flare-alb/834a58e4ec711391"} {"level":"info","ts":1711008037.844712,"logger":"controllers.ingress","msg":"deleting targetGroupBinding","targetGroupBinding":{"namespace":"istio-system","name":"k8s-istiosys-cloudfla-77001729f9"}} {"level":"info","ts":1711008037.9171898,"msg":"deRegistering targets","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:targetgroup/k8s-istiosys-cloudfla-77001729f9/15d7bc13ffdb0974","targets":[
{"AvailabilityZone":null,"Id":"i-049bdb2362160b076","Port":31657}
,{"AvailabilityZone":null,"Id":"i-07c9540ef8de2d9c1","Port":31657},{"AvailabilityZone":null,"Id":"i-0705e01abc70d1d7d","Port":31657},{"AvailabilityZone":null,"Id":"i-083336dfdb9d5db74","Port":31657},{"AvailabilityZone":null,"Id":"i-039f99869440eb9ed","Port":31657},{"AvailabilityZone":null,"Id":"i-0f3e1de4047f936bd","Port":31657},{"AvailabilityZone":null,"Id":"i-00b97651353da83db","Port":31657},{"AvailabilityZone":null,"Id":"i-03f7327fc33f34d70","Port":31657},{"AvailabilityZone":null,"Id":"i-07758d9809dc98a3a","Port":31657},{"AvailabilityZone":null,"Id":"i-0cc122fe832e4a54f","Port":31657},{"AvailabilityZone":null,"Id":"i-07238bec414c06d56","Port":31657},{"AvailabilityZone":null,"Id":"i-0889dc936ea100b3b","Port":31657}]} {"level":"info","ts":1711008038.0244977,"msg":"deRegistered targets","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:targetgroup/k8s-istiosys-cloudfla-77001729f9/15d7bc13ffdb0974"} {"level":"info","ts":1711008038.0562346,"logger":"controllers.ingress","msg":"deleted targetGroupBinding","targetGroupBinding":{"namespace":"istio-system","name":"k8s-istiosys-cloudfla-77001729f9"}} {"level":"info","ts":1711008038.05627,"logger":"controllers.ingress","msg":"deleting targetGroup","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:targetgroup/k8s-istiosys-cloudfla-77001729f9/15d7bc13ffdb0974"} {"level":"info","ts":1711008042.2911835,"logger":"controllers.ingress","msg":"deleted targetGroup","arn":"arn:aws:elasticloadbalancing:us-east-2:566053369253:targetgroup/k8s-istiosys-cloudfla-77001729f9/15d7bc13ffdb0974"} {"level":"info","ts":1711008042.2912066,"logger":"controllers.ingress","msg":"deleting securityGroup","securityGroupID":"sg-094c1bbef9bbec891"} {"level":"info","ts":1711008042.4953787,"logger":"controllers.ingress","msg":"deleted securityGroup","securityGroupID":"sg-094c1bbef9bbec891"} {"level":"info","ts":1711008042.4953992,"logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"istio-system/alb-ingress-cloud-flare-gw"}}}

Definition of the First Ingress:

kind: Gateway
apiVersion: networking.istio.io/v1beta1
metadata:
  name: cloud-flare-gateway
  namespace: isito-system
spec:
  servers:
    - port:
        number: 443
        protocol: HTTPS
        name: https
      hosts:
        - '*'
      tls:
        mode: SIMPLE
        credentialName: istio-ingress-gateway-cert
    - port:
        number: 80
        protocol: HTTP
        name: http
      hosts:
        - '*'
  selector:
    istio: cloud-flare-ingress-gateway

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  annotations:
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/success-codes: '200'
    alb.ingress.kubernetes.io/load-balancer-name: lilly-kubed-dev-cloud-flare-alb
    alb.ingress.kubernetes.io/backend-protocol-version: HTTP1
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/shield-advanced-protection: 'true'
    alb.ingress.kubernetes.io/healthcheck-port: '31343'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: >-
      arn:aws:acm:us-east-2:xxxxxxxxxxxxxxx:certificate/109580a4-d029-4152-9a37-d442cb3114c6
    alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
    alb.ingress.kubernetes.io/inbound-cidrs: '0.0.0.0/0, 10.2.0.0/16'
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
  name: alb-ingress-cloud-flare-gw
  namespace: istio-system
  labels:
    app: lilly-kubed-dev-external-alb-cloud-flare-gw
    app.kubernetes.io/instance: service-mesh
spec:
  tls:
    - hosts:
        - lillydirect.cloudflare-poc.apps.lilly.com
      secretName: istio-ingress-gateway-cert
  rules:
    - host: lillydirect.cloudflare-poc.apps.lilly.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: cloud-flare-ingress-gateway
                port:
                  number: 443

Attachments

Activity

People

Assignee:: Brian Mangoenpawiro

Reporter:: Rod Bryan Domingo

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2024/03/29 6:13 AM

Updated:: 2024/04/03 6:30 AM

Resolved:: 2024/04/03 6:30 AM