Loading...

Type: Bug
Resolution: Done-Errata
Priority: Normal
Fix Version/s: rhos-18.0 Feature Release 1 (Nov 2024)
Affects Version/s: rhos-18.0.0
Component/s: infra-operator
Labels:
- rhos-18-bughunt

Story Points:
5
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Dev Approval:
?
Docs Approval:
?
Fixed in Build:
infra-operator-container-1.0.4-4
PM Approval:
?
QE Approval:
?
Regression:
None
Intelligence Requested:
Market:
Errata Link:
https://errata.engineering.redhat.com/advisory/140345

Severity:
Moderate

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

It seem the memcached replicas field is not properly validated. The value 0 is accepted but leads to a broken keystone and therefore a broken control plane.

$ oc patch openstackcontrolplane/openstack-galera-network-isolation --type='json' -p='[{"op": "replace", "path": "/spec/memcached/templates/memcached/replicas", "value":0}]'

$ openstack endpoint list
Internal Server Error (HTTP 500)
command terminated with exit code 1

[Wed Jun 26 16:14:09.448489 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718] mod_wsgi (pid=22): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
[Wed Jun 26 16:14:09.449499 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718] Traceback (most recent call last):
[Wed Jun 26 16:14:09.449576 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask/app.py", line 2548, in __call__
[Wed Jun 26 16:14:09.449580 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.wsgi_app(environ, start_response)
[Wed Jun 26 16:14:09.449582 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
[Wed Jun 26 16:14:09.449582 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.app(environ, start_response)
[Wed Jun 26 16:14:09.449583 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
[Wed Jun 26 16:14:09.449584 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     resp = self.call_func(req, *args, **kw)
[Wed Jun 26 16:14:09.449585 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
[Wed Jun 26 16:14:09.449586 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.func(req, *args, **kwargs)
[Wed Jun 26 16:14:09.449587 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
[Wed Jun 26 16:14:09.449587 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     response = req.get_response(self.application)
[Wed Jun 26 16:14:09.449588 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1313, in send
[Wed Jun 26 16:14:09.449589 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     status, headers, app_iter = self.call_application(
[Wed Jun 26 16:14:09.449590 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
[Wed Jun 26 16:14:09.449591 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     app_iter = application(self.environ, start_response)
[Wed Jun 26 16:14:09.449592 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 143, in __call__
[Wed Jun 26 16:14:09.449593 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return resp(environ, start_response)
[Wed Jun 26 16:14:09.449594 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
[Wed Jun 26 16:14:09.449595 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     resp = self.call_func(req, *args, **kw)
[Wed Jun 26 16:14:09.449595 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
[Wed Jun 26 16:14:09.449596 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.func(req, *args, **kwargs)
[Wed Jun 26 16:14:09.449597 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
[Wed Jun 26 16:14:09.449598 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     response = req.get_response(self.application)
[Wed Jun 26 16:14:09.449599 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1313, in send
[Wed Jun 26 16:14:09.449601 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     status, headers, app_iter = self.call_application(
[Wed Jun 26 16:14:09.449602 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
[Wed Jun 26 16:14:09.449603 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     app_iter = application(self.environ, start_response)
[Wed Jun 26 16:14:09.449605 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
[Wed Jun 26 16:14:09.449606 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     resp = self.call_func(req, *args, **kw)
[Wed Jun 26 16:14:09.449607 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
[Wed Jun 26 16:14:09.449608 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.func(req, *args, **kwargs)
[Wed Jun 26 16:14:09.449609 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/osprofiler/web.py", line 111, in __call__
[Wed Jun 26 16:14:09.449610 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return request.get_response(self.application)
[Wed Jun 26 16:14:09.449612 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1313, in send
[Wed Jun 26 16:14:09.449613 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     status, headers, app_iter = self.call_application(
[Wed Jun 26 16:14:09.449614 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
[Wed Jun 26 16:14:09.449615 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     app_iter = application(self.environ, start_response)
[Wed Jun 26 16:14:09.449617 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
[Wed Jun 26 16:14:09.449618 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     resp = self.call_func(req, *args, **kw)
[Wed Jun 26 16:14:09.449619 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
[Wed Jun 26 16:14:09.449620 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.func(req, *args, **kwargs)
[Wed Jun 26 16:14:09.449621 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/oslo_middleware/request_id.py", line 58, in __call__
[Wed Jun 26 16:14:09.449623 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     response = req.get_response(self.application)
[Wed Jun 26 16:14:09.449624 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1313, in send
[Wed Jun 26 16:14:09.449625 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     status, headers, app_iter = self.call_application(
[Wed Jun 26 16:14:09.449626 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
[Wed Jun 26 16:14:09.449627 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     app_iter = application(self.environ, start_response)
[Wed Jun 26 16:14:09.449629 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
[Wed Jun 26 16:14:09.449630 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.app(environ, start_response)
[Wed Jun 26 16:14:09.449631 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
[Wed Jun 26 16:14:09.449632 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     resp = self.call_func(req, *args, **kw)
[Wed Jun 26 16:14:09.449634 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
[Wed Jun 26 16:14:09.449634 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.func(req, *args, **kwargs)
[Wed Jun 26 16:14:09.449636 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__
[Wed Jun 26 16:14:09.449653 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     response = req.get_response(self._app)
[Wed Jun 26 16:14:09.449656 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1313, in send
[Wed Jun 26 16:14:09.449657 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     status, headers, app_iter = self.call_application(
[Wed Jun 26 16:14:09.449658 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
[Wed Jun 26 16:14:09.449659 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     app_iter = application(self.environ, start_response)
[Wed Jun 26 16:14:09.449660 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
[Wed Jun 26 16:14:09.449661 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return app(environ, start_response)
[Wed Jun 26 16:14:09.449662 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
[Wed Jun 26 16:14:09.449663 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     response = self.handle_exception(e)
[Wed Jun 26 16:14:09.449663 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
[Wed Jun 26 16:14:09.449664 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return original_handler(e)
[Wed Jun 26 16:14:09.449665 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
[Wed Jun 26 16:14:09.449666 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return original_handler(e)
[Wed Jun 26 16:14:09.449667 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
[Wed Jun 26 16:14:09.449668 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return original_handler(e)
[Wed Jun 26 16:14:09.449669 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   [Previous line repeated 28 more times]
[Wed Jun 26 16:14:09.449670 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask/app.py", line 2525, in wsgi_app
[Wed Jun 26 16:14:09.449670 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     response = self.full_dispatch_request()
[Wed Jun 26 16:14:09.449671 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask/app.py", line 1822, in full_dispatch_request
[Wed Jun 26 16:14:09.449672 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     rv = self.handle_user_exception(e)
[Wed Jun 26 16:14:09.449673 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
[Wed Jun 26 16:14:09.449674 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return original_handler(e)
[Wed Jun 26 16:14:09.449675 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
[Wed Jun 26 16:14:09.449675 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return original_handler(e)
[Wed Jun 26 16:14:09.449676 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
[Wed Jun 26 16:14:09.449677 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return original_handler(e)
[Wed Jun 26 16:14:09.449678 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   [Previous line repeated 28 more times]
[Wed Jun 26 16:14:09.449679 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
[Wed Jun 26 16:14:09.449680 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     rv = self.dispatch_request()
[Wed Jun 26 16:14:09.449681 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
[Wed Jun 26 16:14:09.449681 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
[Wed Jun 26 16:14:09.449683 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 467, in wrapper
[Wed Jun 26 16:14:09.449683 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     resp = resource(*args, **kwargs)
[Wed Jun 26 16:14:09.449684 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask/views.py", line 107, in view
[Wed Jun 26 16:14:09.449685 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
[Wed Jun 26 16:14:09.449686 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
[Wed Jun 26 16:14:09.449687 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     resp = meth(*args, **kwargs)
[Wed Jun 26 16:14:09.449688 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/server/flask/common.py", line 1064, in wrapper
[Wed Jun 26 16:14:09.449688 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return f(*args, **kwargs)
[Wed Jun 26 16:14:09.449689 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/api/auth.py", line 315, in post
[Wed Jun 26 16:14:09.449690 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     token = authentication.authenticate_for_token(auth_data)
[Wed Jun 26 16:14:09.449691 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/api/_shared/authentication.py", line 182, in authenticate_for_token
[Wed Jun 26 16:14:09.449692 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     auth_info = core.AuthInfo.create(auth=auth)
[Wed Jun 26 16:14:09.449693 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/auth/core.py", line 126, in create
[Wed Jun 26 16:14:09.449693 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     auth_info._validate_and_normalize_auth_data(scope_only)
[Wed Jun 26 16:14:09.449695 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/auth/core.py", line 340, in _validate_and_normalize_auth_data
[Wed Jun 26 16:14:09.449695 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     self._validate_and_normalize_scope_data()
[Wed Jun 26 16:14:09.449696 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/auth/core.py", line 290, in _validate_and_normalize_scope_data
[Wed Jun 26 16:14:09.449697 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     project_ref = self._lookup_project(self.auth['scope']['project'])
[Wed Jun 26 16:14:09.449698 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/auth/core.py", line 196, in _lookup_project
[Wed Jun 26 16:14:09.449699 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     domain_ref = self._lookup_domain(project_info['domain'])
[Wed Jun 26 16:14:09.449700 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/auth/core.py", line 171, in _lookup_domain
[Wed Jun 26 16:14:09.449700 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     domain_ref = PROVIDERS.resource_api.get_domain_by_name(
[Wed Jun 26 16:14:09.449701 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/common/manager.py", line 115, in wrapped
[Wed Jun 26 16:14:09.449702 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     __ret_val = __f(*args, **kwargs)
[Wed Jun 26 16:14:09.449703 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "<decorator-gen-28>", line 2, in get_domain_by_name
[Wed Jun 26 16:14:09.449704 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
[Wed Jun 26 16:14:09.449705 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self.get_or_create(
[Wed Jun 26 16:14:09.449706 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 974, in get_or_create
[Wed Jun 26 16:14:09.449707 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     key = self.key_mangler(key)
[Wed Jun 26 16:14:09.449708 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/common/cache/core.py", line 86, in key_mangler
[Wed Jun 26 16:14:09.449708 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     key = '%s:%s' % (key, invalidation_manager.region_id)
[Wed Jun 26 16:14:09.449709 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/common/cache/core.py", line 43, in region_id
[Wed Jun 26 16:14:09.449710 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self._invalidation_region.get_or_create(
[Wed Jun 26 16:14:09.449711 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
[Wed Jun 26 16:14:09.449712 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     with Lock(
[Wed Jun 26 16:14:09.449713 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/lock.py", line 185, in __enter__
[Wed Jun 26 16:14:09.449713 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self._enter()
[Wed Jun 26 16:14:09.449714 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/lock.py", line 87, in _enter
[Wed Jun 26 16:14:09.449715 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     value = value_fn()
[Wed Jun 26 16:14:09.449716 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 977, in get_value
[Wed Jun 26 16:14:09.449717 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     value = self._get_from_backend(key)
[Wed Jun 26 16:14:09.449717 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 1268, in _get_from_backend
[Wed Jun 26 16:14:09.449719 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return cast(CacheReturnType, self.backend.get(key))
[Wed Jun 26 16:14:09.449719 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/keystone/common/cache/_context_cache.py", line 74, in get
[Wed Jun 26 16:14:09.449720 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     value = self.proxied.get(key)
[Wed Jun 26 16:14:09.449721 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/dogpile/cache/backends/memcached.py", line 192, in get
[Wed Jun 26 16:14:09.449722 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     value = self.client.get(key)
[Wed Jun 26 16:14:09.449723 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/pymemcache/client/hash.py", line 361, in get
[Wed Jun 26 16:14:09.449724 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     return self._run_cmd('get', key, None, *args, **kwargs)
[Wed Jun 26 16:14:09.449725 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/pymemcache/client/hash.py", line 326, in _run_cmd
[Wed Jun 26 16:14:09.449726 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     client = self._get_client(key)
[Wed Jun 26 16:14:09.449727 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]   File "/usr/lib/python3.9/site-packages/pymemcache/client/hash.py", line 183, in _get_client
[Wed Jun 26 16:14:09.449727 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718]     raise MemcacheError('All servers seem to be down right now')
[Wed Jun 26 16:14:09.449733 2024] [wsgi:error] [pid 22:tid 55] [remote 10.217.1.180:48718] pymemcache.exceptions.MemcacheError: All servers seem to be down right now
10.217.1.180 - - [26/Jun/2024:16:14:09 +0000] "POST /v3/auth/tokens HTTP/1.1" 500 531 "-" "nova-scheduler keystoneauth1/5.1.2 python-requests/2.25.1 CPython/3.9.19"

At least we should forbid setting the replicas to 0 if this configuration is not supported by keystone.

Moreover even negative replica value is accepted which results in infra-operator errors like:

2024-06-26T16:25:00Z	ERROR	Reconciler error	{"controller": "memcached", "controllerGroup": "memcached.openstack.org", "controllerKind": "Memcached", "Memcached": {"name":"memcached","namespace":"openstack"}, "namespace": "openstack", "name": "memcached", "reconcileID": "d97faf79-ef16-4ec2-83b8-0a6898ef9924", "error": "StatefulSet.apps \"memcached\" is invalid: spec.replicas: Invalid value: -1: must be greater than or equal to 0"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.6/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.6/pkg/internal/controller/controller.go:227

links to

openstack-k8s-operators/infra-operator#253: Validate memcached replicas to be >= 1

RHSA-2024:140345 RHOSO OpenStack Podified operator containers security update

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty