Uploaded image for project: 'OpenShift Migration Toolkit for Containers'
  1. OpenShift Migration Toolkit for Containers
  2. MIG-1339

[MTC] Rsync pod connection hangs after a large period of time migrating data

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Major Major
    • None
    • MTC 1.7.8
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo

      After trying to migrate a Jenkins application using rsync MTC, we've noticed the migration fails because:

      1. The stunnel connection is abruptly closed
      2. The rsync pod is hung, it stopped logging anything at some point.

      We've already increased the rsync pods memory limit to 15 GB to prevent any OOM issues.

      The linked support ticket case summary has a better day-to-day description of the progress of the problem.

      The attachment `0210-all_logs.tar.gz` as the proper logs from the rsync pods in the source and destination project inspection.

      Logs gathered from the source and destination rsync pods:

      • Destination

      stunnel:

      2023-05-15T03:18:47.269491327Z 2023.05.15 03:18:47 LOG6[419]: socket fd: Connection reset by peer (104)
2023-05-15T03:18:47.269491327Z 2023.05.15 03:18:47 LOG6[419]: writesocket: Socket is closed
2023-05-15T03:18:47.269536804Z 2023.05.15 03:18:47 LOG7[419]: Sending close_notify alert
2023-05-15T03:18:47.269536804Z 2023.05.15 03:18:47 LOG7[419]: TLS alert (write): warning: close notify
2023-05-15T03:18:47.269550340Z 2023.05.15 03:18:47 LOG6[419]: SSL_shutdown successfully sent close_notify alert
2023-05-15T03:18:47.269568685Z 2023.05.15 03:18:47 LOG5[419]: Connection closed: 45 byte(s) sent to TLS, 267 byte(s) sent to socket

      rsync:

      2023-05-14T07:39:52.630178753Z 2023/05/14 07:39:52 [381] recv_file_name(logs/slaves/filename3)
2023-05-14T07:39:52.630178753Z 2023/05/14 07:39:52 [381] recv_file_name(logs/slaves/filename4)
2023-05-14T08:13:54.620333315Z 2023/05/14 08:13:54 [391] connect from localhost (::1)
2023-05-14T08:14:04.683572893Z 2023/05/14 08:14:04 [392] connect from localhost (::1)
2023-05-14T08:14:14.750693263Z 2023/05/14 08:14:14 [393] connect from localhost (::1)
2023-05-14T08:14:24.813321395Z 2023/05/14 08:14:24 [394] connect from localhost (::1)
2023-05-14T08:40:11.100822170Z 2023/05/14 08:40:11 [381] rsync: [generator] write error: Connection reset by peer (104)
2023-05-14T08:40:11.100822170Z 2023/05/14 08:40:11 [381] rsync error: error in socket IO (code 10) at io.c(827) [generator=3.1.3]
2023-05-14T08:40:11.100822170Z 2023/05/14 08:40:11 [381] [generator] _exit_cleanup(code=10, file=io.c, line=827): about to call exit(10)
2023-05-14T09:11:22.478839920Z 2023/05/14 09:11:22 [395] connect from localhost (::1)
Source (pod is in status 0/2 Failed)

      stunnel:

      2023-05-14T05:20:02.397958595Z 2023.05.14 05:20:02 LOG5[1]: s_connect: connected 172.28.127.141:443
2023-05-14T05:20:02.397970468Z 2023.05.14 05:20:02 LOG5[1]: Service [rsync] connected remote server from 10.125.15.172:38534
2023-05-14T05:20:02.406034943Z 2023.05.14 05:20:02 LOG5[0]: Connection closed: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
2023-05-14T08:40:11.100860081Z 2023.05.14 08:40:11 LOG3[1]: TLS socket closed (SSL_write) with 18432 unsent byte(s)
2023-05-14T08:40:11.100860081Z 2023.05.14 08:40:11 LOG5[1]: Connection reset: 392863028 byte(s) sent to TLS, 13057487831 byte(s) sent to socket

      rsync:

      2023-05-14T08:40:11.173698725Z recv_file_name(logs/slaves/filename1)
2023-05-14T08:40:11.173705097Z recv_file_name(logs/slaves/filename2)
2023-05-14T08:40:11.173723682Z rsync error: error in socket IO (code 10) at io.c(827) [sender=3.1.3]

              rhn-engineering-dymurray Dylan Murray
              rhn-support-parodrig Pablo Rodriguez Guillamon
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: