Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-27176

ceilometer_agent_compute upstream cannot report metrics

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • None
    • telemetry-operator
    • None
    • Important

      The ceilometer_agent_compute is failing to bring up the prometheus_exporter and it is still trying to publish information into rabbitmq, which is not available, resulting on this error log for each metric:

      2026-03-03 16:19:41.997 12 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.bytes in the context of pollsters
      2026-03-03 16:19:41.998 12 DEBUG ceilometer.compute.pollsters [-] 8b0e58ed-fdfa-451a-bdd9-33f891bbe3d8/network.outgoing.bytes volume: 10800 _stats_to_sample /usr/lib/python3.9/site-packages/ceilometer/compute/pollsters/__init__.py:108
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging [-] Could not send notification to notifications. Payload={'message_id': 'd79b8b11-e8c8-4cde-8b83-44985e6ab9f7', 'publisher_id': 'ceilometer.polling', 'event_type': 'telemetry.polling', 'priority': 'SAMPLE', 'payload': {'samples': [{'source': 'openstack', 'counter_name': 'network.outgoing.bytes', 'counter_type': 'cumulative', 'counter_unit': 'B', 'counter_volume': 10800, 'user_id': 'c3726333660d483a924cf33fc3df6614', 'user_name': None, 'project_id': '0732a5617eb94d98b56a81d58e986f80', 'project_name': None, 'resource_id': 'instance-00000001-8b0e58ed-fdfa-451a-bdd9-33f891bbe3d8-tap352ba156-2b', 'timestamp': '2026-03-03T16:19:41.998053', 'resource_metadata': {'display_name': 'myserver', 'name': 'tap352ba156-2b', 'instance_id': '8b0e58ed-fdfa-451a-bdd9-33f891bbe3d8', 'instance_type': 'm1.tiny', 'host': 'dd2c2f244719d99ff4e330e5fd66a1e36a67c770154edcaa528212c6', 'instance_host': 'edpm-compute-1.ctlplane.example.com', 'flavor': {'id': 'ad7fa9fb-ae2e-4f31-9599-54227084f59e', 'name': 'm1.tiny', 'vcpus': 1, 'ram': 512, 'disk': 1, 'ephemeral': 0, 'swap': 0}, 'status': 'active', 'state': 'running', 'task_state': '', 'image': {'id': '560641b9-2165-4930-a677-1ee339283268'}, 'image_ref': '560641b9-2165-4930-a677-1ee339283268', 'image_ref_url': None, 'architecture': 'x86_64', 'os_type': 'hvm', 'vcpus': 1, 'memory_mb': 512, 'disk_gb': 1, 'ephemeral_gb': 0, 'root_gb': 1, 'mac': 'fa:16:3e:4e:de:cf', 'fref': None, 'parameters': {'interfaceid': None, 'bridge': None}, 'vnic_name': 'tap352ba156-2b'}, 'message_id': 'c8c39ea4-171c-11f1-9882-525400aa5210', 'monotonic_time': 8162.482661471, 'message_signature': '078e3adabdf3017455c316cca1552435710113d5045d05010ced5e83b4391457'}]}, 'timestamp': '2026-03-03 16:19:41.998416', '_unique_id': '32f221aa97cb4723ac346374efc27dde'}: kombu.exceptions.OperationalError: [Errno 111] Connection refused
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     yield
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/connection.py", line 433, in _ensure_connection
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     return retry_over_time(
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     return fun(*args, **kwargs)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/connection.py", line 877, in _connection_factory
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self._connection = self._establish_connection()
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/connection.py", line 812, in _establish_connection
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     conn = self.transport.establish_connection()
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     conn.connect()
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/amqp/connection.py", line 323, in connect
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self.transport.connect()
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/amqp/transport.py", line 129, in connect
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self._connect(self.host, self.port, self.connect_timeout)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/amqp/transport.py", line 184, in _connect
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self.sock.connect(sa)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging ConnectionRefusedError: [Errno 111] Connection refused
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging The above exception was the direct cause of the following exception:
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/notify/messaging.py", line 78, in notify
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self.transport._send_notification(target, ctxt, message,
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 134, in _send_notification
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self._driver.send_notification(target, ctxt, message, version,
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 694, in send_notification
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     return self._send(target, ctxt, message,
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 653, in _send
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     with self._get_connection(rpc_common.PURPOSE_SEND, retry) as conn:
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 605, in _get_connection
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     return rpc_common.ConnectionContext(self._connection_pool,
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/common.py", line 423, in __init__
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self.connection = connection_pool.get(retry=retry)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/pool.py", line 98, in get
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     return self.create(retry=retry)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/pool.py", line 135, in create
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     return self.connection_cls(self.conf, self.url, purpose, retry=retry)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 826, in __init__
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self.ensure_connection()
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 957, in ensure_connection
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self.connection.ensure_connection(
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/connection.py", line 381, in ensure_connection
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self._ensure_connection(*args, **kwargs)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/connection.py", line 433, in _ensure_connection
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     return retry_over_time(
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     self.gen.throw(typ, value, traceback)
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.9/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging     raise ConnectionError(str(exc)) from exc
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging kombu.exceptions.OperationalError: [Errno 111] Connection refused
      2026-03-03 16:19:41.999 12 ERROR oslo_messaging.notify.messaging 

              rhn-engineering-jlarriba Juan Larriba
              rhn-engineering-jlarriba Juan Larriba
              rhos-observability-telemetry
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: