Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-1330

On OCP 4.20, 3scale-kourier-gateway crashing due to "socket(2) failed, got error: Too many open files"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • 1.37.0
    • 1.36.0, 1.37.0
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      On OCP 4.20, running any larger test scenario

      the 3scale-kourier-gateway crashes on

      [2025-09-23 14:34:15.231][12][critical][assert] [external/envoy/source/common/network/socket_interface_impl.cc:72] assert failure: SOCKET_VALID(result.return_value_). Details: socket(2) failed, got error: Too many open files
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3ba6b0e000000001
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 3daafe6fc90bd0c0a3b0138169e7e3e111816645/1.26.8/OSSM 2.5.12-1/RELEASE/OpenSSL
      [symbolize_elf.inc : 1000] RAW: /proc/self/task/1/maps: errno=24
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x7f44983df990]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #1: [0x556beb2316b0]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #2: [0x556beb17f75e]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #3: [0x556beb17bc05]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #4: [0x556beb1735cf]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #5: [0x556beb159cc3]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #6: [0x556beadd208e]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #7: [0x556beadd18e6]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #8: [0x556beadd2765]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #9: [0x556beadafb71]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #10: [0x556beadaf942]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #11: [0x556beadb0f6a]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #12: [0x556beadb137b]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #13: [0x556beadc3664]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #14: [0x556beadc64d4]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #15: [0x556beadb50c2]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #16: [0x556beb04c607]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #17: [0x556beb0662a0]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #18: [0x556beb05248d]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #19: [0x556beb0dd8c9]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #20: [0x556beaf5cd7c]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #21: [0x556beaf8c2ca]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #22: [0x556beaf892d8]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #23: [0x556beaf88fd8]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #24: [0x556beb20e21b]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #25: [0x556beaf87096]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #26: [0x556beaf8691f]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #27: [0x556beaf8c11f]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #28: [0x556beaf5780c]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #29: [0x556beb1815b0]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #30: [0x556beb17a6a2]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #31: [0x556beb17818e]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #32: [0x556beb15ccbf]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #33: [0x556beb15de52]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #34: [0x556beb2476ed]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #35: [0x556beb246171]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #36: [0x556beaa82091]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #37: [0x556beb3136b5]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #38: [0x7f44983d51ca]
      ActiveStream 0x556befd9e000, stream_id_: 7505110563027602138&filter_manager_: 
        FilterManager 0x556befd9e0a8, state_.has_1xx_headers_: 0
        filter_manager_callbacks_.requestHeaders(): 
          ':authority', 'sender6-ksnk-dn-0.apps.soak-1-36-40.ci-psi.rhocf-dev.net'
          ':path', '/healthz'
          ':method', 'GET'
          ':scheme', 'http'
          'user-agent', 'Knative-Ingress-Probe'
          'k-network-probe', 'probe'
          'accept-encoding', 'gzip'
          'x-forwarded-proto', 'http'
          'x-request-id', '3791ddec-1382-4173-831f-d5b7955af608'
          'knative-serving-namespace', 'ksnk-dn-0'
          'knative-serving-revision', 'sender6-00002'
          'k-network-hash', '52713468f3cd3c53623d84edc6b42f4fd8ddbf25a7e5919ace1c93477166b4c5'
        filter_manager_callbacks_.requestTrailers():   null
        filter_manager_callbacks_.responseHeaders():   null
        filter_manager_callbacks_.responseTrailers():   null
        &streamInfo(): 
          StreamInfoImpl 0x556befd9e1d8, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: 1, health_check_request_: 0, route_name_: (ksnk-dn-0/sender6).Rules[1].Paths[/]    upstream_info_: 
            UpstreamInfoImpl 0x556befb68170, upstream_connection_id_: null
          OverridableRemoteConnectionInfoSetterStreamInfo 0x556befd9e1d8, remoteAddress(): 10.129.6.14:56454, directRemoteAddress(): 10.129.6.14:56454, localAddress(): 10.129.4.19:8090
      Http1::ConnectionImpl 0x556bf015a008, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 1, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Done, current_header_field_: , current_header_value_: 
      active_request_: 
      , request_url_: null, response_encoder_.local_end_stream_: 0
      absl::get<RequestHeaderMapPtr>(headers_or_trailers_): null
      current_dispatching_buffer_ front_slice length: 197 contents: "GET /healthz HTTP/1.1\r\nHost: sender6-ksnk-dn-0.apps.soak-1-36-40.ci-psi.rhocf-dev.net\r\nUser-Agent: Knative-Ingress-Probe\r\nK-Network-Hash: override\r\nK-Network-Probe: probe\r\nAccept-Encoding: gzip\r\n\r\n"
      ConnectionImpl 0x556bef8ed520, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
      socket_: 
        ListenSocketImpl 0x556bef4dad00, transport_protocol_: raw_buffer
        connection_info_provider_: 
          ConnectionInfoSetterImpl 0x556bee2c4100, remote_address_: 10.129.6.14:56454, direct_remote_address_: 10.129.6.14:56454, local_address_: 10.129.4.19:8090, server_name_: 
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 3daafe6fc90bd0c0a3b0138169e7e3e111816645/1.26.8/OSSM 2.5.12-1/RELEASE/OpenSSL
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x7f44983df990]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #1: [0x556beb2316b0]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #2: [0x556beb17f75e]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #3: [0x556beb17bc05]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #4: [0x556beb1735cf]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #5: [0x556beb159cc3]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #6: [0x556beadd208e]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #7: [0x556beadd18e6]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #8: [0x556beadd2765]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #9: [0x556beadafb71]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #10: [0x556beadaf942]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #11: [0x556beadb0f6a]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #12: [0x556beadb137b]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #13: [0x556beadc3664]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #14: [0x556beadc64d4]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #15: [0x556beadb50c2]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #16: [0x556beb04c607]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #17: [0x556beb0662a0]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #18: [0x556beb05248d]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #19: [0x556beb0dd8c9]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #20: [0x556beaf5cd7c]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #21: [0x556beaf8c2ca]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #22: [0x556beaf892d8]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #23: [0x556beaf88fd8]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #24: [0x556beb20e21b]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #25: [0x556beaf87096]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #26: [0x556beaf8691f]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #27: [0x556beaf8c11f]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #28: [0x556beaf5780c]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #29: [0x556beb1815b0]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #30: [0x556beb17a6a2]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #31: [0x556beb17818e]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #32: [0x556beb15ccbf]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #33: [0x556beb15de52]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #34: [0x556beb2476ed]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #35: [0x556beb246171]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #36: [0x556beaa82091]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #37: [0x556beb3136b5]
      [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #38: [0x7f44983d51ca]
      Our FatalActions triggered a fatal signal.
       

      The problem seems to be caused by the new `ulimit -n` (max. number of open file descriptors) default value on OCP 4.20, which is just 1024  (compared to 1048576 on previous OCP versions)

              Unassigned Unassigned
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: