Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-13328

collectd and ceilometer logs are spammed with libvirt.libvirtError because they can't get stats for DPDK interfaces

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Rejected
    • Important

      Our NPSS customer in EMEA is preparing to move RHOSP 17.1 OSPDo deployment to production. While checking if environment is affected by common problems, we have found spam of the following errors in /var/log/containers/ceilometer/compute.log [1] and /var/log/containers/stdouts/collectd.log [2] on some compute node. Errors seem to be related to VM with DPDK interfaces and are also logged in /var/log/containers/libvirt/virtqemud.log [3].

      It looks like there is a bug that doesn't allow collectd and ceilometer to properly process error raised by libvirt for this case.

      [1]

      2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters [-] Could not get network.incoming.packets.error events for INSTANCE_UUID: internal error: Interface doesn't have any statistics: libvirt.libvirtError: internal error: Interface doesn't have any statistics
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters Traceback (most recent call last):
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters   File "/usr/lib/python3.9/site-packages/ceilometer/compute/pollsters/__init__.py", line 135, in get_samples
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters     polled_time, result = self._inspect_cached(
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters   File "/usr/lib/python3.9/site-packages/ceilometer/compute/pollsters/__init__.py", line 100, in _inspect_cached
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters     result = list(result)
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters   File "/usr/lib/python3.9/site-packages/ceilometer/compute/virt/libvirt/inspector.py", line 117, in inspect_vnics
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters     dom_stats = domain.interfaceStats(name)
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters   File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1721, in interfaceStats
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters     raise libvirtError('virDomainInterfaceStats() failed')
2025-01-14 13:01:38.714 10 ERROR ceilometer.compute.pollsters libvirt.libvirtError: internal error: Interface doesn't have any statistics

      [2]

      /var/log/containers/stdouts/collectd.log
2025-01-14T00:01:15.392020588+01:00 stderr F libvirt:  error : internal error: Interface doesn't have any statistics

      [3]

      /var/log/containers/libvirt/virtqemud.log
2025-01-13 23:01:38.314+0000: 69849: error : virNetDevOpenvswitchInterfaceStats:409 : internal error: Interface doesn't have any statistics

              Unassigned Unassigned
              rhn-support-astupnik Alex Stupnikov
              rhos-conplat-observability
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: