Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-81279

[GSS] domain-notify.sock socket is missing from virt-handler pod

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • CNV Virtualization
    • None
    • Quality / Stability / Reliability
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • Important
    • Customer Reported
    • None

      Description of problem:

      Virt-launcer is unable to communicate to the virt-handler pod:
      
      2026-03-04T17:50:52.134256300Z {"component":"virt-launcher","level":"info","msg":"failed to dial notify socket: /var/run/kubevirt/domain-notify-pipe.sock","pos":"client.go:149","reason":"context deadline exceeded","timestamp":"2026-03-04T17:50:52.134106Z"}
      2026-03-04T17:50:52.134256300Z {"component":"virt-launcher","level":"error","msg":"Failed to connect to notify server","pos":"client.go:209","reason":"context deadline exceeded","timestamp":"2026-03-04T17:50:52.134195Z"}
      2026-03-04T17:50:52.790657036Z {"component":"virt-launcher","level":"info","msg":"Monitoring loop: rate 1s start timeout 4m46s","pos":"monitor.go:209","timestamp":"2026-03-04T17:50:52.790540Z"}
      2026-03-04T17:50:53.791805366Z {"component":"virt-launcher","level":"info","msg":"Found PID for 1cld-powervc_99299: 81","pos":"monitor.go:170","timestamp":"2026-03-04T17:50:53.791645Z"}
      2026-03-04T17:50:54.135336169Z {"component":"virt-launcher","level":"info","msg":"failed to dial notify socket: /var/run/kubevirt/domain-notify-pipe.sock","pos":"client.go:149","reason":"context deadline exceeded","timestamp":"2026-03-04T17:50:54.135190Z"}
      
      because the virt-handler pod is missing the domain-notify socket file:
      
      2026-03-04T18:57:50.027428929Z {"component":"virt-handler","level":"error","msg":"error connecting to domain-notify.sock for proxy connection","pos":"vm.go:396","reason":"dial unix /var/run/kubevirt/domain-notify.sock: connect: no such file or directory","timestamp":"2026-03-04T18:57:50.027262Z"}
      2026-03-04T18:57:50.066272944Z {"component":"virt-handler","level":"error","msg":"error connecting to domain-notify.sock for proxy connection","pos":"vm.go:396","reason":"dial unix /var/run/kubevirt/domain-notify.sock: connect: no such file or directory","timestamp":"2026-03-04T18:57:50.066129Z"}
      
      During the time of this issue, we've noticed issues OOMkiller killing pods. However, triggering OOMkills has yield no valid results, the socket file never goes missing. This bug is being created to understand why this socket file is going missing and in what part of the chain this happens. 

      Version-Release number of selected component (if applicable):

      kubevirt-hyperconverged-operator.v4.19.18
      OCP: 4.19.13

      How reproducible:

      N/A

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

       

      Expected results:

       

      Additional info:

      Workaround is outlined in the KCS https://access.redhat.com/solutions/7007566 which is to restart the virt-handler pods.
      

              ibezukh Igor Bezukh
              rhn-support-kelwhite Kelson White
              Kedar Bidarkar Kedar Bidarkar
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: