Uploaded image for project: 'Migration Toolkit for Virtualization'
  1. Migration Toolkit for Virtualization
  2. MTV-1220

forklift-controller , inventory container was restarted during 5K migration test.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 2.6.2
    • Inventory
    • False
    • None
    • False
    • Important

      During testing migration of 5000 VMs using 10 plans (running in parallel), inventory container was restarted with ErrorCode: 2.

      pod describe:
        inventory:
          Container ID:  cri-o://bad27778a966555a77b6de6c16c54fa2f333a72e6c5ada0fbfd3b730b1b09050
          State:          Running
            Started:      Fri, 21 Jun 2024 06:11:51 +0000
          Last State:     Terminated
            Reason:       Error
            Exit Code:    2
            Started:      Thu, 20 Jun 2024 10:40:41 +0000
            Finished:     Fri, 21 Jun 2024 06:11:47 +0000
          Ready:          True
          Restart Count:  1

      inventory log:

      panic: concurrent write to websocket connection
              panic: concurrent write to websocket connection

      goroutine 4943150 [running]:
      github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc004688300, 0x1, {0x0?, 0x29b1560?, 0xc00824b808?})
              /remote-source/app/vendor/github.com/gorilla/websocket/conn.go:632 +0x4b8
      github.com/gorilla/websocket.(*messageWriter).Close(0x30?)
              /remote-source/app/vendor/github.com/gorilla/websocket/conn.go:746 +0x35
      github.com/gorilla/websocket.(*Conn).beginMessage(0xc00aa1fce0, 0xc0044ec870, 0x1)
              /remote-source/app/vendor/github.com/gorilla/websocket/conn.go:493 +0x47

      main log:

      sxs-plan7-2xhgb","dv":"mtv26-5000vms-8esxs/mtv26-5000vms-8esxs-plan7-vm-5515-8vq2c","vm":" id:vm-5515 name:'dsl-4g-thick-fc-vm-3906' "}

      {"level":"info","ts":"2024-06-21 06:10:53.375","logger":"plan|rmlck","msg":"Itinerary transition","plan":\{"name":"mtv26-5000vms-8esxs-plan7","namespace":"openshift-mtv"}

      ,"migration":"openshift-mtv/mtv26-5000vms-8esxs-plan7-2xhgb","current phase":"CreateDataVolumes","next phase":"AllocateDisks"}

      {"level":"info","ts":"2024-06-21 06:11:07.520","logger":"watch","msg":"event: error.","id":9,"error":"unknown error event received"} {"level":"debug","ts":"2024-06-21 06:11:07.533","logger":"watch","msg":"event: started.","id":25} {"level":"info","ts":"2024-06-21 06:11:27.918","logger":"watch","msg":"event: error.","id":7,"error":"unknown error event received"} {"level":"debug","ts":"2024-06-21 06:11:28.024","logger":"watch","msg":"event: started.","id":26} {"level":"info","ts":"2024-06-21 06:11:44.622","logger":"watch","msg":"event: error.","id":25,"error":"unknown error event received"} {"level":"debug","ts":"2024-06-21 06:11:44.817","logger":"watch","msg":"event: started.","id":27} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":26,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":11,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":12,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":13,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":8,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":27,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"}

        1. Conroller_Restart_inventory.log.xz
          860 kB
          David Vaanunu
        2. Conroller_Restart.log.xz
          848 kB
          David Vaanunu

            Unassigned Unassigned
            dvaanunu@redhat.com David Vaanunu
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: