Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-26450

Getting http 500 internal server error due to "ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds"

XMLWordPrintable

    • Important
    • None

      Description of problem:
      Getting the following error on default 2 workers 16 threads Puma configurations when sending more than 16 query requests at the same time.
      ----------------------------------------------
      ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use
      ----------------------------------------------

      The error is gone after settings the connection pools to 32 (number of worker * number of threads)

      /usr/share/foreman/config/database.yml

      1. Database is managed by foreman::database::postgresql
        production:
        adapter: postgresql
        database: foreman
        username: foreman
        password: *********
        pool: 32 <==============

      Steps to Reproduce:
      1. On any remote host or Satellite itself, run the command to send many concurrent requests to the web server using irb console

      irb
      require 'rest_client'
      50.times { Thread.new

      { begin; RestClient::Resource.new("https://satellite.example.com/api/v2/hosts?installed_package_name=kernel&page=1&per_page=200", user: "admin", password: "pass", timeout: 3600, open_timeout: 3600, verify_ssl: OpenSSL::SSL::VERIFY_NONE).get; rescue StandardError => e; p e.message; end }

      }

      2. On Satellite, tail the /var/log/foreman/production.log

      Actual results:
      ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use

      Requests getting 500 internal server error

      Expected results:
      No error

      Additional info:
      I thought each worker process has a separate connection pool but somehow it seems like it is not. Not sure.

              rhn-engineering-sshtein Shimon Shtein
              rhn-support-hyu Hao Chang Yu
              Pablo Méndez Hernández Pablo Méndez Hernández
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: