Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-36493

Proxied subscription-manager traffic to Satellite fails under high concurrency

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 6.17.1
    • Candlepin, Performance
    • False
    • Important
    • sat-proton
    • None
    • None
    • None
    • None

      Description:

      Commands such as $(subscription-manager identity) or $(subscription-manager repos --enable=<repo_id>) can fail if executed on scale and if there's a proxy between Satellite and hosts:

      Proxy error: unable to connect to <proxy_host>:<proxy_port>: Remote end closed connection without response.

      Managing systems via Ansible becomes difficult because tasks involving subscription-manager commands intermittently fail on some target hosts. When running batches of 80 hosts, roughly 20 hosts fail per execution. To mitigate this, administrators are forced to reduce the batch size.

      This behavior can be reliably reproduced with just a single host.

      Prerequisites for reproducer:

      • Have Satellite 6.17.
      • Have a squid proxy.
      • Have a host configured to use this proxy and registered to Satellite.

      How to reproduce:

      To simulate high concurrency, execute $(subscription-manager identity) in loop:

      # cat subman_identity.sh
      --- 
      for i in {1..80}; do
        subscription-manager identity &
      done
      wait
      ---

      Most iterations succeed, but there are some that fail, e.g.:

      system identity: b06a730f-c58d-46b9-a8fc-c78a466a85f7
      name: jsenkyri-client1.sysmgmt.lan
      org name: Default Organization
      org ID: Default_OrganizationProxy error: unable to connect to jsenkyri-squid.sysmgmt.lan:3128: Remote end closed connection without response.

      At some point during the execution, the connection is dropped. While Satellite logs do not report any errors, the httpd access logs indicate the connection was terminated mid-execution. The logs for failed hosts are missing some GET requests that are present for hosts that succeeded.

      Full trace from rhsm.log:

      2025-08-06 16:13:23,429 [ERROR] subscription-manager:10002:MainThread @cli.py:71 - Error: Unable to generate a new identity for the system
      2025-08-06 16:13:23,312 [ERROR] subscription-manager:9999:MainThread @cli.py:72 - Proxy error at jsenkyri-squid.sysmgmt.lan:3128: Remote end closed connection without response
      Traceback (most recent call last):
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1049, in _make_request
          response = conn.getresponse()
        File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse
          response.begin()
        File "/usr/lib64/python3.9/http/client.py", line 320, in begin
          version, status, reason = self._read_status()
        File "/usr/lib64/python3.9/http/client.py", line 289, in _read_status
          raise RemoteDisconnected("Remote end closed connection without"
      http.client.RemoteDisconnected: Remote end closed connection without response
      During handling of the above exception, another exception occurred:
      Traceback (most recent call last):
        File "/usr/lib64/python3.9/site-packages/subscription_manager/cli_command/identity.py", line 96, in _do_command
          consumer = self.cp.getConsumer(consumerid)
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1800, in getConsumer
          return self.conn.request_get(method, description=_("Fetching consumer keys"))
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1365, in request_get
          result: Dict[str, Any] = self._request(
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1153, in _request
          result, response = self._make_request(
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1089, in _make_request
          raise NoValidEntitlement(
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1082, in _make_request
          raise ProxyException(hostname=self.proxy_hostname, port=self.proxy_port, exc=err)
      rhsm.connection.ProxyException: Proxy error at jsenkyri-squid.sysmgmt.lan:3128: Remote end closed connection without response
      2025-08-06 16:13:23,688 [ERROR] subscription-manager:10002:MainThread @cli.py:72 - Proxy error at jsenkyri-squid.sysmgmt.lan:3128: Remote end closed connection without response
      Traceback (most recent call last):
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1049, in _make_request
          response = conn.getresponse()
        File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse
          response.begin()
        File "/usr/lib64/python3.9/http/client.py", line 320, in begin
          version, status, reason = self._read_status()
        File "/usr/lib64/python3.9/http/client.py", line 289, in _read_status
          raise RemoteDisconnected("Remote end closed connection without"
      http.client.RemoteDisconnected: Remote end closed connection without response
      During handling of the above exception, another exception occurred:
      Traceback (most recent call last):
        File "/usr/lib64/python3.9/site-packages/subscription_manager/cli_command/identity.py", line 96, in _do_command
          consumer = self.cp.getConsumer(consumerid)
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1800, in getConsumer
          return self.conn.request_get(method, description=_("Fetching consumer keys"))
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1365, in request_get
          result: Dict[str, Any] = self._request(
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1153, in _request
          result, response = self._make_request(
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1089, in _make_request
          raise NoValidEntitlement(
        File "/usr/lib64/python3.9/site-packages/rhsm/connection.py", line 1082, in _make_request
          raise ProxyException(hostname=self.proxy_hostname, port=self.proxy_port, exc=err)
      rhsm.connection.ProxyException: Proxy error at jsenkyri-squid.sysmgmt.lan:3128: Remote end closed connection without response
      

      Additional information:

      • This is not reproducible when the system(s) are registered directly to RHSM (subscription.rhsm.redhat.com).
      • This is not reproducible when there's no proxy between Satellite and host(s).

       

              rhn-engineering-pablomh Pablo Méndez Hernández
              rhn-support-jsenkyri Jan Senkyrik
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: