On OpenShift 4.14, An ingress object was created with an invalid certificate configuration [1]. This ingress object lead to a route creation which was automatically ingested by the router default pods (expected). However, the route itself was invalid as a result of the problematic certificate. This caused the router pods to fail reloads, leading to inconsistent backend state and a failure to update the haproxy.config in response to backend changes - leading to a partial outage.

issue is likely that haproxy is not expecting the private key to include Bag Attributes bundled with the Root and Intermediate cert.

Outage was resolved after route removal, which allowed the haproxy router pods to reload successfully on their own + self-recover.

    4.14.31

    one time - not re-attempted due to production cluster state

    1. Create an ingress object, and tie a tls certificate and key combo in that includes bag attributes bundled with root + intermediate certificate (or similar invalid cert state)
    2. Observe route gets created and accepted by haproxy router pods
    3. Observe log output [2] generated in haproxy router pods
    4. Observe new pods not contacted by router pods or deleted pods still attempting to have traffic forwarded (router-reload failures).
    

    partial outage caused by router-default handling issue

haproxy should validate and reject routes that are invalid, not stall reload processing while attempting to log error state. Should evict the problem route instead.     

sosreport + log bundle available below.

    [1] certificate details - redacted

tls.crt:
-----BEGIN CERTIFICATE-----
"redacted"
-----END CERTIFICATE-----


tls.key
Bag Attributes
    friendlyName: 2024_ocp4_prod_<redacted>
    localKeyID: 54 69 6D 65 20 31 37 32 32 39 37 31 35 33 36 37 34 34
    1.3.18.0.2.28.24: IBM_SDK_JAVA_8_PKCS12
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
REDACTED
-----END PRIVATE KEY-----
Bag Attributes
    friendlyName: <redacted>ca1
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
    1.3.18.0.2.28.24: IBM_SDK_JAVA_8_PKCS12
subject=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Root CA 1
issuer=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Root CA 1
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: <redacted>ca8
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
    1.3.18.0.2.28.24: IBM_SDK_JAVA_8_PKCS12
subject=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Issuing CA 8
issuer=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Root CA 1
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----




[2] haproxy logs:

~~~

[NOTICE]   (90908) : path to executable is /usr/sbin/haproxy
[ALERT]    (90908) : config : parsing [/var/lib/haproxy/conf/haproxy.config:132] : 'bind unix@/var/lib/haproxy/run/haproxy-sni.sock' in section 'frontend' : 'crt-list' : inconsistencies between private key and certificate loaded '/var/lib/haproxy/router/certs/eside:pushnotificationpersistentservice-qrjwl.pem'.
[ALERT]    (90908) : config : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT]    (90908) : config : Fatal errors found in configuration.
E0201 00:36:22.933698       1 limiter.go:165] error reloading router: exit status 1
[NOTICE]   (90912) : haproxy version is 2.6.13-234aa6d
[NOTICE]   (90912) : path to executable is /usr/sbin/haproxy
[ALERT]    (90912) : config : parsing [/var/lib/haproxy/conf/haproxy.config:132] : 'bind unix@/var/lib/haproxy/run/haproxy-sni.sock' in section 'frontend' : 'crt-list' : inconsistencies between private key and certificate loaded '/var/lib/haproxy/router/certs/eside:pushnotificationpersistentservice-qrjwl.pem'.
[ALERT]    (90912) : config : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT]    (90912) : config : Fatal errors found in configuration.