Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-8595

Quay does not start if Splunk has an outage

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      Quay is configured to ignore audit logging failures by setting `ALLOW_WITHOUT_STRICT_LOGGING: True` as per PROJQUAY-7116 .

      There is an issue in Splunk, where an internal Splunk certificate, for communication between Splunk and Mongodb, which is where the tokens are stored, is expired.

      Quay keeps working, but if some pod is restarted, it will not start correctly and won't work, returning the following errors:

      Traceback (most recent call last):
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 320, in wrapper
          return request_fun(self, *args, **kwargs)
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 79, in new_f
          val = f(*args, **kwargs)
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 727, in get
          response = self.http.get(path, all_headers, **query)
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 1254, in get
          return self.request(url, { 'method': "GET", 'headers': headers })
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 1326, in request
          raise HTTPError(response)
      splunklib.binding.HTTPError: HTTP 401 Unauthorized -- call not properly authenticated
      
      
      
      Traceback (most recent call last):
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 262, in _handle_auth_error
          yield
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 330, in wrapper
          return request_fun(self, *args, **kwargs)
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 79, in new_f
          val = f(*args, **kwargs)
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 727, in get
          response = self.http.get(path, all_headers, **query)
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 1254, in get
          return self.request(url, { 'method': "GET", 'headers': headers })
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 1326, in request
          raise HTTPError(response)
      splunklib.binding.HTTPError: HTTP 401 Unauthorized -- call not properly authenticated
      
      
      
      Traceback (most recent call last):
        File "/quay-registry/boot.py", line 14, in <module>
          from app import app
        File "/quay-registry/app.py", line 338, in <module>
          logs_model.configure(app.config)
        File "/quay-registry/data/logs_model/__init__.py", line 67, in configure
          logs_model.initialize(_LOG_MODELS[model_name](**model_config))
        File "/quay-registry/data/logs_model/splunk_logs_model.py", line 30, in __init__
          self._logs_producer.initialize(SplunkLogsProducer(**splunk_config))
        File "/quay-registry/data/logs_model/logs_producer/splunk_logs_producer.py", line 56, in __init__
          self.index = service.indexes[index_prefix]
        File "/app/lib/python3.9/site-packages/splunklib/client.py", line 1376, in __getitem__
          response = self.get(key)
        File "/app/lib/python3.9/site-packages/splunklib/client.py", line 1804, in get
          return super(Collection, self).get(name, owner, app, sharing, **query)
        File "/app/lib/python3.9/site-packages/splunklib/client.py", line 862, in get
          return self.service.get(path,
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 330, in wrapper
          return request_fun(self, *args, **kwargs)
        File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
          self.gen.throw(typ, value, traceback)
        File "/app/lib/python3.9/site-packages/splunklib/binding.py", line 265, in _handle_auth_error
          raise AuthenticationError(msg, he)
      splunklib.binding.AuthenticationError: Authentication Failed! If session token is used, it seems to have been expired. 

      Even if Quay cannot authenticate against Splunk, it should start and work normally.

              Unassigned Unassigned
              rhn-support-rauferna Raul Fernandez
              luffy zhang luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: