Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-80993

virtctl console: connection drops (1006) or hangs when VM runs long-running container / high serial output (v1.8.x)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • CNV Perf/Scale
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • None

      Description of problem:

      With KubeVirt server v1.8.0-beta.0-331-g8711eeaca0 (and virtctl client 1.8.0), the serial console connection becomes unusable when a long-running process runs inside the VM (e.g. a container with a long or heavy workload)

      Version-Release number of selected component (if applicable):

      OCP 4.22.0-ec.2
      CNV 4.22.0-17
      Client: v1.8.0-beta.0 (or matching 1.8.x)
      Server: v1.8.0-beta.0-331-g8711eeaca0

      How reproducible:

      Create a VM that boots and has cloud-init run a long-running workload (e.g. podman run ... that runs for several minutes and prints to stdout), or log into the VM and start such a command manually.

      Steps to Reproduce:

      runcmd:
        - echo "Flooding serial..."
        - bash -c 'for i in $(seq 1 50000); do echo "line $i"; done'
        - echo "Done"
      Connect to the VM serial console with virtctl:
      virtctl console -n <namespace> <vm_name>
      and/or capture it:
      virtctl console -n <namespace> <vm_name> > /tmp/vm.log. 

      Actual results:

      Observe: the console disconnects with 1006 / unexpected EOF, and/or the capture file stays empty or stops updating (console appears stuck), while the workload inside the VM continues and produces output when accessed interactivel

      Expected results:

      VM should finish the run w/o any error

      Additional info:

      - Works on OCP 4.21 with KubeVirt 1.7.1 (same VM image and workload).
      - Fails on OCP 4.22 with KubeVirt 1.8.x.
      - Both client and server are 1.8.x (no version skew). virtctl is used to capture the serial console for CI (benchmark-runner); the VM runs a container that can run for 10+ minutes with continuous serial output.
      Reducing kernel log level in the VM and redirecting container stderr to a file reduce serial noise but do not fix the disconnect or stuck capture.

              dagur@redhat.com Daniel Gur
              ebattat@redhat.com Eli Battat
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Created:
                Updated: