-
Bug
-
Resolution: Unresolved
-
Blocker
-
1.36.0, 1.37.0
-
None
-
None
-
False
-
-
False
-
-
On OCP 4.20, running any larger test scenario
the 3scale-kourier-gateway crashes on
[2025-09-23 14:34:15.231][12][critical][assert] [external/envoy/source/common/network/socket_interface_impl.cc:72] assert failure: SOCKET_VALID(result.return_value_). Details: socket(2) failed, got error: Too many open files [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3ba6b0e000000001 [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers): [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 3daafe6fc90bd0c0a3b0138169e7e3e111816645/1.26.8/OSSM 2.5.12-1/RELEASE/OpenSSL [symbolize_elf.inc : 1000] RAW: /proc/self/task/1/maps: errno=24 [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x7f44983df990] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #1: [0x556beb2316b0] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #2: [0x556beb17f75e] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #3: [0x556beb17bc05] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #4: [0x556beb1735cf] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #5: [0x556beb159cc3] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #6: [0x556beadd208e] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #7: [0x556beadd18e6] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #8: [0x556beadd2765] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #9: [0x556beadafb71] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #10: [0x556beadaf942] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #11: [0x556beadb0f6a] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #12: [0x556beadb137b] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #13: [0x556beadc3664] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #14: [0x556beadc64d4] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #15: [0x556beadb50c2] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #16: [0x556beb04c607] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #17: [0x556beb0662a0] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #18: [0x556beb05248d] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #19: [0x556beb0dd8c9] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #20: [0x556beaf5cd7c] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #21: [0x556beaf8c2ca] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #22: [0x556beaf892d8] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #23: [0x556beaf88fd8] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #24: [0x556beb20e21b] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #25: [0x556beaf87096] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #26: [0x556beaf8691f] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #27: [0x556beaf8c11f] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #28: [0x556beaf5780c] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #29: [0x556beb1815b0] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #30: [0x556beb17a6a2] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #31: [0x556beb17818e] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #32: [0x556beb15ccbf] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #33: [0x556beb15de52] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #34: [0x556beb2476ed] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #35: [0x556beb246171] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #36: [0x556beaa82091] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #37: [0x556beb3136b5] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #38: [0x7f44983d51ca] ActiveStream 0x556befd9e000, stream_id_: 7505110563027602138&filter_manager_: FilterManager 0x556befd9e0a8, state_.has_1xx_headers_: 0 filter_manager_callbacks_.requestHeaders(): ':authority', 'sender6-ksnk-dn-0.apps.soak-1-36-40.ci-psi.rhocf-dev.net' ':path', '/healthz' ':method', 'GET' ':scheme', 'http' 'user-agent', 'Knative-Ingress-Probe' 'k-network-probe', 'probe' 'accept-encoding', 'gzip' 'x-forwarded-proto', 'http' 'x-request-id', '3791ddec-1382-4173-831f-d5b7955af608' 'knative-serving-namespace', 'ksnk-dn-0' 'knative-serving-revision', 'sender6-00002' 'k-network-hash', '52713468f3cd3c53623d84edc6b42f4fd8ddbf25a7e5919ace1c93477166b4c5' filter_manager_callbacks_.requestTrailers(): null filter_manager_callbacks_.responseHeaders(): null filter_manager_callbacks_.responseTrailers(): null &streamInfo(): StreamInfoImpl 0x556befd9e1d8, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: 1, health_check_request_: 0, route_name_: (ksnk-dn-0/sender6).Rules[1].Paths[/] upstream_info_: UpstreamInfoImpl 0x556befb68170, upstream_connection_id_: null OverridableRemoteConnectionInfoSetterStreamInfo 0x556befd9e1d8, remoteAddress(): 10.129.6.14:56454, directRemoteAddress(): 10.129.6.14:56454, localAddress(): 10.129.4.19:8090 Http1::ConnectionImpl 0x556bf015a008, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 1, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Done, current_header_field_: , current_header_value_: active_request_: , request_url_: null, response_encoder_.local_end_stream_: 0 absl::get<RequestHeaderMapPtr>(headers_or_trailers_): null current_dispatching_buffer_ front_slice length: 197 contents: "GET /healthz HTTP/1.1\r\nHost: sender6-ksnk-dn-0.apps.soak-1-36-40.ci-psi.rhocf-dev.net\r\nUser-Agent: Knative-Ingress-Probe\r\nK-Network-Hash: override\r\nK-Network-Probe: probe\r\nAccept-Encoding: gzip\r\n\r\n" ConnectionImpl 0x556bef8ed520, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576 socket_: ListenSocketImpl 0x556bef4dad00, transport_protocol_: raw_buffer connection_info_provider_: ConnectionInfoSetterImpl 0x556bee2c4100, remote_address_: 10.129.6.14:56454, direct_remote_address_: 10.129.6.14:56454, local_address_: 10.129.4.19:8090, server_name_: [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0 [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers): [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 3daafe6fc90bd0c0a3b0138169e7e3e111816645/1.26.8/OSSM 2.5.12-1/RELEASE/OpenSSL [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x7f44983df990] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #1: [0x556beb2316b0] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #2: [0x556beb17f75e] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #3: [0x556beb17bc05] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #4: [0x556beb1735cf] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #5: [0x556beb159cc3] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #6: [0x556beadd208e] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #7: [0x556beadd18e6] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #8: [0x556beadd2765] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #9: [0x556beadafb71] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #10: [0x556beadaf942] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #11: [0x556beadb0f6a] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #12: [0x556beadb137b] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #13: [0x556beadc3664] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #14: [0x556beadc64d4] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #15: [0x556beadb50c2] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #16: [0x556beb04c607] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #17: [0x556beb0662a0] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #18: [0x556beb05248d] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #19: [0x556beb0dd8c9] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #20: [0x556beaf5cd7c] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #21: [0x556beaf8c2ca] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #22: [0x556beaf892d8] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #23: [0x556beaf88fd8] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #24: [0x556beb20e21b] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #25: [0x556beaf87096] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #26: [0x556beaf8691f] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #27: [0x556beaf8c11f] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #28: [0x556beaf5780c] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #29: [0x556beb1815b0] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #30: [0x556beb17a6a2] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #31: [0x556beb17818e] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #32: [0x556beb15ccbf] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #33: [0x556beb15de52] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #34: [0x556beb2476ed] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #35: [0x556beb246171] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #36: [0x556beaa82091] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #37: [0x556beb3136b5] [2025-09-23 14:34:15.231][12][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #38: [0x7f44983d51ca] Our FatalActions triggered a fatal signal.
The problem seems to be caused by the new `ulimit -n` (max. number of open file descriptors) default value on OCP 4.20, which is just 1024 (compared to 1048576 on previous OCP versions)
- is caused by
-
OCPBUGS-62095 "ulimit -n" just 1024 (instead of 1048576)
-
- Verified
-