Loading...

Type: Bug
Resolution: Duplicate
Priority: Minor
Fix Version/s: None
Affects Version/s: 4.12.z
Component/s: Networking / router
Labels:

Severity:
Low
Regression:
No
Story Points:
1
Sprint:
Sprint 254
sprint_count:
1
Blocked:
False
Blocked Reason:

Hide

None

Show
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

The customer created the route with some wrong path by mistake due to which route wasn't accessible and after when the router pods were restarted to troubleshoot the issue, it was observed that router pods weren't able to come up until the problematic route was deleted.

Create the route with the wrong configuration.

$ oc new-project httpd

$ oc new-app registry.redhat.io/rhel8/httpd-24

$ cat route
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations:
    haproxy.router.openshift.io/rewrite-target: /
    meta.helm.sh/release-name: httpd
    meta.helm.sh/release-namespace: httpd
  labels:
  name: httpd
spec:
  host: httpd.example.com
  path: /%!(EXTRA string=main, string=/, string=deploytest, string=/)
  port:
    targetPort: 8080
  tls:
    termination: edge
  to:
    kind: Service
    name: httpd-24
    weight: 100
  wildcardPolicy: None

$ oc create -f route
route.route.openshift.io/httpd created

$ oc get route
NAME    HOST/PORT           PATH                                                            SERVICES   PORT   TERMINATION   WILDCARD
httpd   httpd.example.com   /%!(EXTRA string=main, string=/, string=deploytest, string=/)   httpd-24   8080   edge          None

Delete the router pods and verify the status.

$ oc project openshift-ingress
$ oc get pod
NAME                             READY   STATUS    RESTARTS   AGE
router-default-7b4689cf8-7n8vh   1/1     Running   0          2d4h
router-default-7b4689cf8-ckd2l   1/1     Running   0          2d4h
$ oc delete pod --all
pod "router-default-7b4689cf8-7n8vh" deleted
pod "router-default-7b4689cf8-ckd2l" deleted
$ oc get pod
NAME                             READY   STATUS        RESTARTS   AGE
router-default-7b4689cf8-848gb   0/1     Running       0          62s
router-default-7b4689cf8-f9vjs   0/1     Running       0          62s
$ oc logs router-default-7b4689cf8-848gb
I0520 17:36:00.241674       1 template.go:437] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: 431a6e667025931c68b4f747a224af29edf356f6\nversionFromGit: 4.0.0-429-g431a6e66\ngitTreeState: clean\nbuildDate: 2024-03-05T17:57:00Z\n"
I0520 17:36:00.243278       1 metrics.go:156] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936"
I0520 17:36:00.248309       1 router.go:200] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy"
I0520 17:36:00.248371       1 router.go:282] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s"
I0520 17:36:00.248733       1 router.go:352] template "msg"="watching for changes" "path"="/etc/pki/tls/private"
I0520 17:36:00.248794       1 router.go:269] router "msg"="router is including routes in all namespaces"
E0520 17:36:00.356482       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
E0520 17:36:00.371024       1 limiter.go:165] error reloading router: exit status 1
[NOTICE] 140/173600 (18) : haproxy version is 2.2.24-26b8015
[NOTICE] 140/173600 (18) : path to executable is /usr/sbin/haproxy
[ALERT] 140/173600 (18) : parsing [/var/lib/haproxy/conf/haproxy.config:215] : error detected in backend 'be_edge_http:httpd:httpd' while parsing 'http-request replace-path' rule : expects exactly 2 arguments <match-regex> and <replace-format>.
[ALERT] 140/173600 (18) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
I0520 17:36:01.091232       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:02.090440       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:03.091149       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:04.090742       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
E0520 17:36:04.869008       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
I0520 17:36:05.090450       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
E0520 17:36:05.357634       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
E0520 17:36:05.369771       1 limiter.go:165] error reloading router: exit status 1
[NOTICE] 140/173605 (22) : haproxy version is 2.2.24-26b8015
[NOTICE] 140/173605 (22) : path to executable is /usr/sbin/haproxy
[ALERT] 140/173605 (22) : parsing [/var/lib/haproxy/conf/haproxy.config:215] : error detected in backend 'be_edge_http:httpd:httpd' while parsing 'http-request replace-path' rule : expects exactly 2 arguments <match-regex> and <replace-format>.
[ALERT] 140/173605 (22) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
I0520 17:36:06.091299       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:07.090469       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:08.091544       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:09.090720       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:10.091309       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:11.090734       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:12.089933       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:13.090078       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
E0520 17:36:14.036157       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
I0520 17:36:14.090355       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:15.090198       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:16.090612       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:17.090871       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:18.090539       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:19.090625       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:20.090450       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:21.090660       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:22.090578       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:23.090149       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure
I0520 17:36:24.090908       1 healthz.go:261] backend-http check failed: healthz
[-]backend-http failed: backend reported failure

The issue will be resolved only after the problematic route is deleted.

$ oc project httpd
$ oc delete route httpd
route.route.openshift.io "httpd" deleted

$ oc get pod -n openshift-ingress
NAME                             READY   STATUS    RESTARTS       AGE
router-default-7b4689cf8-848gb   1/1     Running   1 (2m2s ago)   5m47s
router-default-7b4689cf8-f9vjs   1/1     Running   2 (3s ago)     5m47s

There should be some check or mechanism to avoid the router failure if any route with wrong syntax is created.

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates