Uploaded image for project: 'Multiple Architecture Enablement'
  1. Multiple Architecture Enablement
  2. MULTIARCH-4617

Libvirt builds failing with "failed to connect to the hypervisor" error

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • Multi-Arch CI
    • None
    • False
    • None
    • False
    • NEW
    • NEW

      We are observing failures in libvirt runs scheduled on Mihawk C155F2U35 due to below error. More than 8 jobs have failed due to this error from yesterday and today.

      error: failed to connect to the hypervisor error: unable to connect to server at 'C155F2U35:16509': Connection refused

      we did some checks and saw that apici_build01 service was running fine and enough space is available,
      but only error I observed was when I checked the libvirtd service status although it is active but there is a error due a stale VM/network entry,

      [root@C155F2U35 ~]# systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2024-03-20 05:40:03 EDT; 1 months 2 days ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 2936 (libvirtd)
    Tasks: 20 (limit: 32768)
   Memory: 163.4M
   CGroup: /system.slice/libvirtd.service
           └─2936 /usr/sbin/libvirtd --listen

Apr 21 08:23:03 C155F2U35 libvirtd[2936]: Failed to get host CPU cache info
Apr 21 08:23:03 C155F2U35 libvirtd[2936]: Failed to get host CPU cache info
Apr 21 08:42:17 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.13 52:54:00:26:cf:8d
Apr 21 08:42:17 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.13 52:54:00:26:cf:8d libvirt-ppc64le-2-1-3-4z247-master-2
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.12 52:54:00:62:a1:07
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.12 52:54:00:62:a1:07 libvirt-ppc64le-2-1-3-4z247-master-1
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.11 52:54:00:9a:3e:13
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.11 52:54:00:9a:3e:13 libvirt-ppc64le-2-1-3-4z247-master-0
Apr 21 08:48:32 C155F2U35 dnsmasq[2218883]: exiting on receipt of SIGTERM
Apr 21 08:48:34 C155F2U35 libvirtd[2936]: End of file while reading data: Input/output error

      so when I closely checked libvirtd service observed this line "DHCPACK(br2-1) 192.168.1.13 52:54:00:26:cf:8d libvirt-ppc64le-2-1-3-4z247-master-2" and checked when and by which build this VM was created and then got to know that it was created by the ppc auxillary build run on 21-04-2024 at 5:30 PM IST, so using virsh command I checked if there are any VM's or networks are present with the above id but could not find any, but these VM's id is only observed in libvirtd service status.
      We have checked the timestamp of this build and got to know that builds which have run on C155F2U35 mihawk after this build all are hitting this error "failed to connect to the hypervisor" so we think this might be the potential cause of this
      error.
      So we tried with restarting the libvirtd service, but this did not resolve the issue.
      so I have attached journalctl logs of libvirt service

      Apr 21 08:21:30 C155F2U35 libvirtd[2936]: Cannot connect to QEMU guest agent for libvirt-ppc64le-2-1-3-4z247-worker-0-hgz4j
Apr 21 08:21:31 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.51 3a:82:7c:a5:b0:6c
Apr 21 08:21:31 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.51 3a:82:7c:a5:b0:6c libvirt-ppc64le-2-1-3-4z247-worker-0->
Apr 21 08:21:46 C155F2U35 dnsmasq-dhcp[2218883]: DHCPDISCOVER(br2-1) b2:27:50:a5:1a:fa
Apr 21 08:21:46 C155F2U35 dnsmasq-dhcp[2218883]: DHCPOFFER(br2-1) 192.168.1.52 b2:27:50:a5:1a:fa
Apr 21 08:21:46 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.52 b2:27:50:a5:1a:fa
Apr 21 08:21:46 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.52 b2:27:50:a5:1a:fa libvirt-ppc64le-2-1-3-4z247-worker-0->
Apr 21 08:21:48 C155F2U35 dnsmasq-dhcp[2218883]: DHCPDISCOVER(br2-1) 3a:82:7c:a5:b0:6c
Apr 21 08:21:48 C155F2U35 dnsmasq-dhcp[2218883]: DHCPOFFER(br2-1) 192.168.1.51 3a:82:7c:a5:b0:6c
Apr 21 08:21:48 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.51 3a:82:7c:a5:b0:6c
Apr 21 08:21:48 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.51 3a:82:7c:a5:b0:6c libvirt-ppc64le-2-1-3-4z247-worker-0->
Apr 21 08:23:03 C155F2U35 libvirtd[2936]: Failed to get host CPU cache info
Apr 21 08:23:03 C155F2U35 libvirtd[2936]: Failed to get host CPU cache info
Apr 21 08:23:03 C155F2U35 libvirtd[2936]: Failed to get host CPU cache info
Apr 21 08:42:17 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.13 52:54:00:26:cf:8d
Apr 21 08:42:17 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.13 52:54:00:26:cf:8d libvirt-ppc64le-2-1-3-4z247-master-2
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.12 52:54:00:62:a1:07
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.12 52:54:00:62:a1:07 libvirt-ppc64le-2-1-3-4z247-master-1
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPREQUEST(br2-1) 192.168.1.11 52:54:00:9a:3e:13
Apr 21 08:42:22 C155F2U35 dnsmasq-dhcp[2218883]: DHCPACK(br2-1) 192.168.1.11 52:54:00:9a:3e:13 libvirt-ppc64le-2-1-3-4z247-master-0
Apr 21 08:48:32 C155F2U35 dnsmasq[2218883]: exiting on receipt of SIGTERM
Apr 21 08:48:34 C155F2U35 libvirtd[2936]: End of file while reading data: Input/output error
Apr 22 06:37:28 C155F2U35 systemd[1]: Stopping Virtualization daemon...
Apr 22 06:37:28 C155F2U35 systemd[1]: libvirtd.service: Succeeded.
Apr 22 06:37:28 C155F2U35 systemd[1]: Stopped Virtualization daemon.
Apr 22 06:37:28 C155F2U35 systemd[1]: Starting Virtualization daemon...
Apr 22 06:37:28 C155F2U35 systemd[1]: Started Virtualization daemon.

      which is indicating "End of file while reading data: Input/output error" error/warning, so now we need to check why this error is occurring. 
      Some extra logs
      tried restarting apici_build01 service and observed the output in sshd-bastion

      Connection closed by ::1 port 34722
Transferred: sent 103580, received 92188 bytes
Closing connection to ::1 port 34722
Connection from ::1 port 51166 on ::1 port 2222
Connection from ::1 port 51164 on ::1 port 2222
Connection from ::1 port 51180 on ::1 port 2222
Connection from ::1 port 51188 on ::1 port 2222
Postponed publickey for root from ::1 port 51188 ssh2 [preauth]
Accepted publickey for root from ::1 port 51188 ssh2: RSA SHA256:mIIKdHWRT+nybUtH07OtCqxeVj5qPmStfFKw6kSSqHo
Connection closed by ::1 port 51164 [preauth]
Connection closed by ::1 port 51166 [preauth]
Connection closed by ::1 port 51180 [preauth]
      [root@C155F2U35 ~]# df -h Filesystem
[root@C155F2U35 ~]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
devtmpfs                         484G     0  484G   0% /dev
tmpfs                            510G     0  510G   0% /dev/shm
tmpfs                            510G  291M  510G   1% /run
tmpfs                            510G     0  510G   0% /sys/fs/cgroup
/dev/mapper/rhel_c155f2u35-root  1.5T   14G  1.5T   1% /
/dev/nvme0n1p1                  1014M  243M  772M  24% /boot
tmpfs                            102G     0  102G   0% /run/user/0

       

              shgokul Shilpa Gokul
              sgudaji1 Suraj Gudaji
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: