During testing migration of 5000 VMs using 10 plans (running in parallel), inventory container was restarted with ErrorCode: 2.
pod describe:
inventory:
Container ID: cri-o://bad27778a966555a77b6de6c16c54fa2f333a72e6c5ada0fbfd3b730b1b09050
State: Running
Started: Fri, 21 Jun 2024 06:11:51 +0000
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 20 Jun 2024 10:40:41 +0000
Finished: Fri, 21 Jun 2024 06:11:47 +0000
Ready: True
Restart Count: 1
inventory log:
panic: concurrent write to websocket connection
panic: concurrent write to websocket connection
goroutine 4943150 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc004688300, 0x1, {0x0?, 0x29b1560?, 0xc00824b808?})
/remote-source/app/vendor/github.com/gorilla/websocket/conn.go:632 +0x4b8
github.com/gorilla/websocket.(*messageWriter).Close(0x30?)
/remote-source/app/vendor/github.com/gorilla/websocket/conn.go:746 +0x35
github.com/gorilla/websocket.(*Conn).beginMessage(0xc00aa1fce0, 0xc0044ec870, 0x1)
/remote-source/app/vendor/github.com/gorilla/websocket/conn.go:493 +0x47
main log:
sxs-plan7-2xhgb","dv":"mtv26-5000vms-8esxs/mtv26-5000vms-8esxs-plan7-vm-5515-8vq2c","vm":" id:vm-5515 name:'dsl-4g-thick-fc-vm-3906' "}
{"level":"info","ts":"2024-06-21 06:10:53.375","logger":"plan|rmlck","msg":"Itinerary transition","plan":\{"name":"mtv26-5000vms-8esxs-plan7","namespace":"openshift-mtv"},"migration":"openshift-mtv/mtv26-5000vms-8esxs-plan7-2xhgb","current phase":"CreateDataVolumes","next phase":"AllocateDisks"}
{"level":"info","ts":"2024-06-21 06:11:07.520","logger":"watch","msg":"event: error.","id":9,"error":"unknown error event received"} {"level":"debug","ts":"2024-06-21 06:11:07.533","logger":"watch","msg":"event: started.","id":25} {"level":"info","ts":"2024-06-21 06:11:27.918","logger":"watch","msg":"event: error.","id":7,"error":"unknown error event received"} {"level":"debug","ts":"2024-06-21 06:11:28.024","logger":"watch","msg":"event: started.","id":26} {"level":"info","ts":"2024-06-21 06:11:44.622","logger":"watch","msg":"event: error.","id":25,"error":"unknown error event received"} {"level":"debug","ts":"2024-06-21 06:11:44.817","logger":"watch","msg":"event: started.","id":27} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":26,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":11,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":12,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":13,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":8,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"} {"level":"info","ts":"2024-06-21 06:11:50.233","logger":"watch","msg":"event: error.","id":27,"error":"websocket: close 1006 (abnormal closure): unexpected EOF"}