Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-47773

Route spec.path that have space or # causes HaProxy error and ingress degraded

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 5
    • Important
    • None
    • None
    • Rejected
    • NE Sprint 265, NI&D Sprint 266, NI&D Sprint 267, NI&D Sprint 270, NI&D Sprint 271, NI&D Sprint 272, NI&D Sprint 273, NI&D Sprint 275, NI&D Sprint 276
    • 9
    • Customer Escalated
    • +
    • In Progress
    • Bug Fix
    • Hide
      Cause: The Route API validation allowed a route to specify a path that contained a ' ' space character or '#' octothorpe character, and OpenShift router did not filter these characters out when writing the HAProxy configuration.

      Consequence: Users were able to create routes with invalid paths that caused HAProxy to refuse to start.

      Fix: The Route API validation now prohibits space and octothorpe characters in path values.

      Result: Users can no longer create a route with either of these characters in the path.
      Show
      Cause: The Route API validation allowed a route to specify a path that contained a ' ' space character or '#' octothorpe character, and OpenShift router did not filter these characters out when writing the HAProxy configuration. Consequence: Users were able to create routes with invalid paths that caused HAProxy to refuse to start. Fix: The Route API validation now prohibits space and octothorpe characters in path values. Result: Users can no longer create a route with either of these characters in the path.

      Description of problem:

      A route with the following spec.path with a space (/route-admission-test1 / test1) or # (/route-admission-test!#) and  (annotations:     haproxy.router.openshift.io/rewrite-target: /)
      can broke haproxy and cause ingress degraded.
      
      ~~~
      kind: Route
      apiVersion: route.openshift.io/v1
      metadata:
        name: hello-openshift
        namespace: route-admission-test1
        annotations:
          haproxy.router.openshift.io/rewrite-target: /
      spec:
        host: example-route-admission-test1.apps.xxxxx.japaneast.aroapp.io
        path: /route-admission-test1 / test1
        to:
          kind: Service
          name: hello-openshift
          weight: 100
        tls:
          termination: edge
          insecureEdgeTerminationPolicy: Allow
        wildcardPolicy: None
      ~~~

      Version-Release number of selected component (if applicable):

      4.14 (other version could also have same issue)

      How reproducible:

      Create a route with the haproxy.router.openshift.io/rewrite-target annotation and which spec.path have space like
       /route-admission-test1 / test1 
      or with #
       /route-admission#

      Steps to Reproduce:

      1.oc new-project route-admission-test1     
      
      2. oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json     
      
      3.oc expose pod/hello-openshift
      
      4. Create route like below 
      
      kind: Route
      apiVersion: route.openshift.io/v1
      metadata:
        name: hello-openshift
        namespace: route-admission-test1
        annotations:
          haproxy.router.openshift.io/rewrite-target: /
      spec:
        host: example-route-admission-test1.apps.xxxxx.japaneast.aroapp.io
        path: /route-admission-test1 / test1
        to:
          kind: Service
          name: hello-openshift
          weight: 100
        tls:
          termination: edge
          insecureEdgeTerminationPolicy: Allow
        wildcardPolicy: None
      
      
      
      5. delete ingress pods and then check ingress status
      
      
      
      $ oc get pods -n openshift-ingress
      NAME                              READY   STATUS    RESTARTS       AGE
      router-default-5fbb99fbf6-d4mvf   0/1     Running   2 (109s ago)   7m21s
      router-default-5fbb99fbf6-mfd2f   1/1     Running   0              19m
      
      6. check logs
      
      $ oc logs pods/router-default-5fbb99fbf6-d4mvf -n openshift-ingress
      I0106 04:01:13.664076       1 template.go:559] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: b3af193f191b8d20c0b7bac328cb2c397d5c2c27\nversionFromGit: 4.0.0-462-gb3af193f\ngitTreeState: clean\nbuildDate: 2024-03-06T00:47:12Z\n"
      I0106 04:01:13.667376       1 metrics.go:156] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936"
      I0106 04:01:13.671519       1 router.go:210] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy"
      I0106 04:01:13.671608       1 router.go:294] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s"
      I0106 04:01:13.672088       1 router.go:364] template "msg"="watching for changes" "path"="/etc/pki/tls/private"
      I0106 04:01:13.672164       1 router.go:269] router "msg"="router is including routes in all namespaces" 
      E0106 04:01:13.781453       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
      E0106 04:01:13.798536       1 limiter.go:165] error reloading router: exit status 1
      [NOTICE]   (18) : haproxy version is 2.6.13-234aa6d
      [NOTICE]   (18) : path to executable is /usr/sbin/haproxy
      [ALERT]    (18) : config : parsing [/var/lib/haproxy/conf/haproxy.config:381] : error detected in backend 'be_edge_http:route-admission-test1:hello-openshift' while parsing 'http-request replace-path' rule : expects exactly 2 arguments <match-regex> and <replace-format>.
      [ALERT]    (18) : config : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
      
      .........
      [-]backend-http failed: backend reported failure
      I0106 04:03:14.376923       1 healthz.go:261] backend-http check failed: healthz
      [-]backend-http failed: backend reported failure
      I0106 04:03:14.392670       1 template.go:828] router "msg"="Shutdown requested, waiting 45s for new connections to cease" 
      E0106 04:03:24.418696       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
      jayu@jayu-thinkpadp15vgen1:~/Desktop/Jan3-2025$ [-]backend-http failed: backend reported failure
      I0106 04:03:14.392670       1 template.go:828] router "msg"="Shutdown requested, waiting 45s for new connections to cease" 
      E0106 04:03:24.418696       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
      
      
      
      7 check ingress operator
      
      $ oc get co
      NAME                                       VERSION               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      aro                                        v20241111.00-hotfix   True       ..........
      ingress                                    4.14.16               True        True          False      28m     ingresscontroller "default" is progressing: IngressControllerProgressing: One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 1 of 2 updated replica(s) are available......
      insights                                  

      Actual results:

      ingress degraded when there are space or # in spec.path of route yaml  

      Expected results:

      when create route there should be some check prevent customer to setup space or # like (path: /route-admission-test1 / test1) when create Route     

      Additional info:

      1) There is a similar bug like 
      https://issues.redhat.com/browse/OCPBUGS-27741 
      also shows same issue but the engineer suggested to create new bug for space issue , that is why I create this new bug.
      
      
      2) for the reproduce test case , when change the route's spec.path from (/route-admission-test1 / test1) to (/route-admission-test1/test1) will solve the issue , but we should have that check to prevent that happens.
      
      kind: Route
      apiVersion: route.openshift.io/v1
      metadata:
        name: hello-openshift
        namespace: route-admission-test1
        annotations:
          haproxy.router.openshift.io/rewrite-target: /
      spec:
        host: example-route-admission-test1.apps.xxxxx.japaneast.aroapp.io
        path: /route-admission-test1/test1
        to:
          kind: Service
          name: hello-openshift
          weight: 100
        tls:
          termination: edge
          insecureEdgeTerminationPolicy: Allow
        wildcardPolicy: None    

       

              rh-ee-alsyed Ali Syed
              rhn-support-jayu Jacob Yu
              None
              Ricardo Pchevuzinske Katz
              Shudi Li Shudi Li
              None
              Votes:
              3 Vote for this issue
              Watchers:
              20 Start watching this issue

                Created:
                Updated:
                Resolved: