Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-43095

[release-4.13] Look for an option to identify double header encoding which cause disruption with HAProxy 2.6

XMLWordPrintable

    • Important
    • None
    • 5
    • NE Sprint 260, NE Sprint 261
    • 2
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, upgrading to OpenShift Container Platform 4.14 caused service disruptions because of duplicate Transfer-Encoding headers that were configured at the application level. This resulted in rejected responses, including a gateway error, which affected customer service. With this release, cluster administrators can use a procedure to detect applications that send duplicate Transfer-Encoding headers before upgrading to OpenShift Container Platform 4.13. Any issues can be corrected before causing service disruption.
      ====

      ### Release Note: Detecting Duplicate Transfer-Encoding Headers Before Upgrading to OpenShift 4.14

      #### Summary:
      A critical issue has been identified where upgrading to OpenShift 4.14 (which includes HAProxy 2.6) causes service disruptions due to duplicate `Transfer-Encoding` headers configured at the application level. This behaviour, compliant with RFC 7230, results in rejected responses with a 502 Bad Gateway error, impacting customer services.

      To address this, cluster administrators can now use a procedure to proactively detect applications that are sending duplicate `Transfer-Encoding` headers before upgrading. This allows administrators to mitigate the issue in advance, preventing service disruption.

      #### Key Points:
      - **Problem**: When upgrading to OpenShift 4.14, HAProxy 2.6 enforces strict RFC 7230 compliance, rejecting requests with multiple `Transfer-Encoding` headers. This causes disruptions in services that rely on misconfigured applications.
      - **Solution**: Cluster administrators are advised to follow the procedure outlined in [Red Hat Solution 7055002](https://access.redhat.com/solutions/7055002) to identify and rectify duplicate `Transfer-Encoding` headers before upgrading. This proactive step helps avoid the potential for widespread service failures.

      #### Next Steps:
      To assist with this, the following procedure is recommended:
      1. **Pre-Upgrade Detection**: Use the newly introduced method to scan for duplicate `Transfer-Encoding` headers in routes and applications running in the cluster before upgrading to OpenShift 4.14.
      2. **Metrics and Monitoring**: Check for metrics related to duplicate headers (`haproxy_backend_duplicate_te_header_total`) to identify problematic routes.
      3. **Mitigation**: Address applications with duplicate headers prior to upgrading, ensuring compliance with the new HAProxy 2.6 restrictions.

      Following this approach will help prevent service disruption post-upgrade, allowing a smoother transition to OpenShift 4.14.
      Show
      Previously, upgrading to OpenShift Container Platform 4.14 caused service disruptions because of duplicate Transfer-Encoding headers that were configured at the application level. This resulted in rejected responses, including a gateway error, which affected customer service. With this release, cluster administrators can use a procedure to detect applications that send duplicate Transfer-Encoding headers before upgrading to OpenShift Container Platform 4.13. Any issues can be corrected before causing service disruption. ==== ### Release Note: Detecting Duplicate Transfer-Encoding Headers Before Upgrading to OpenShift 4.14 #### Summary: A critical issue has been identified where upgrading to OpenShift 4.14 (which includes HAProxy 2.6) causes service disruptions due to duplicate `Transfer-Encoding` headers configured at the application level. This behaviour, compliant with RFC 7230, results in rejected responses with a 502 Bad Gateway error, impacting customer services. To address this, cluster administrators can now use a procedure to proactively detect applications that are sending duplicate `Transfer-Encoding` headers before upgrading. This allows administrators to mitigate the issue in advance, preventing service disruption. #### Key Points: - **Problem**: When upgrading to OpenShift 4.14, HAProxy 2.6 enforces strict RFC 7230 compliance, rejecting requests with multiple `Transfer-Encoding` headers. This causes disruptions in services that rely on misconfigured applications. - **Solution**: Cluster administrators are advised to follow the procedure outlined in [Red Hat Solution 7055002]( https://access.redhat.com/solutions/7055002 ) to identify and rectify duplicate `Transfer-Encoding` headers before upgrading. This proactive step helps avoid the potential for widespread service failures. #### Next Steps: To assist with this, the following procedure is recommended: 1. **Pre-Upgrade Detection**: Use the newly introduced method to scan for duplicate `Transfer-Encoding` headers in routes and applications running in the cluster before upgrading to OpenShift 4.14. 2. **Metrics and Monitoring**: Check for metrics related to duplicate headers (`haproxy_backend_duplicate_te_header_total`) to identify problematic routes. 3. **Mitigation**: Address applications with duplicate headers prior to upgrading, ensuring compliance with the new HAProxy 2.6 restrictions. Following this approach will help prevent service disruption post-upgrade, allowing a smoother transition to OpenShift 4.14.
    • Bug Fix
    • Done

      Description of problem:

      With the HAProxy 2.6, the double header enconding configured at the application level is generating disruption on criticals customer services.
Cluster Admin is requesting if is possible to get a procedure to identify those application header from OCP in order to anticipate the potencial issue reported at solution 7055002 [1] before upgrading the cluster to OCP 4.14
This is a blocker for biggest account as Support Exception are being requested to deploy older version of HAProxy on  OCP 4.14 clusters

      Version-Release number of selected component (if applicable):

          4.14

      How reproducible:

      Reported internally

      Steps to Reproduce:

          1. With an  OCP 4.14 route:
$ curl -kv --header 'Content-Type: application/json' --data '{"code": 0}' -H "transfer-encoding: chunked" -H "transfer-encoding: chunked" --resolve httpd-ex.apps.example.com:80:10.74.208.127 http://httpd-ex.apps.example.com     
...
> Content-Type: application/json 
> transfer-encoding: chunked 
> transfer-encoding: chunked 
> 
< HTTP/1.1 400 Bad request 
< Content-length: 90 
< Cache-Control: no-cache 
< Connection: close 
< Content-Type: text/html   

      Actual results:

          502/400 error results

      Expected results:

      200 OK

       

      [1] https://access.redhat.com/solutions/7055002

              amcdermo@redhat.com Andrew McDermott
              rhn-support-pescorza Pamela Lizeth Escorza Gil
              Shudi Li Shudi Li
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: