Watcher channel closes after some time

Description of problem:

In this PR, we started using watcher channels to wait for the job finished event from the periodic and on-demand data gathering jobs from IO.
However, as stated in this comment, part of maintaining a watcher is to re-establish it at the last received resource version whenever this channel closes.

This issue is currently causing flakiness in our test suite as the on-demand data gathering job is created, when the job is about to finish, the watcher channel closes, which is causing the datagather instance associated with the job to never have the insightsReport updated. Therefore the tests fail.

Version-Release number of selected component (if applicable):

How reproducible:

Sometimes. Very hard to reproduce as it might have to do with the API resyncing the watcher's cache .

Steps to Reproduce:

    1.Create a data gathering job
    2.You may see a log saying "watcher channel was closed unexpectedly"

    

Actual results:

The DataGather instance will not be updated with the insightsReport    

Expected results:

When the job finishes, the archive is uploaded to ingress and the report is downloaded from the external data pipeline. This report should appear in the DataGather instance.

Additional info:

It's possible but flaky to reproduce with on-demand data gathering jobs but I've seen it happen with periodic ones as well.