we lack visibility in build failures and this leads to broken builds for... months

• 5.0.5 is our main branch, the interrupt catcher is pinged when:

• pac CI fails on each PR or after merge
• check-patch, check-merged, build-pipeline, release-pipeline fail
• CVP builds fail
• acceptance tests fail

• failures on release branches are handled by release captains:
• pac CI fails on each PR or after merge
• check-patch, check-merged, build-pipeline, release-pipeline fail
• CVP builds fail
• acceptance tests fail

• we also need cron based alerting on release branches branches that checks and pings every day

when an MR is opened on a release branch of gitlab/p12n:

• interrupt catcher is notified for the 5.0.5 branch, release captains are notified for a release branch
• tests are run including: check-patch and only relevant upstream tests are run so we don't consume resources unnecessarily and also save time
• 🔥 if things fail, slack notifications are sent to release captains

when an MR merged on a release branch:

•  pac runs:
• upstream tests
• acceptance tests (i.e. release tests)

• cpaas runs
• check-patch, check-merged, build-pipeline, release-pipeline
• CVP builds

• 🔥 if any tests fail including cpaas, slack notifications are sent to release captains

⏳ move from cron based model to a watcher based model

currently, when:

• sources are updated in downstream
• when the dashboard is updated
• nightlies are run

all of this is done via cronjobs which is wasteful of the resources on the cluster, and updates are also delayed to the dashboard and repositories

we need to move to a watcher based model, or a pub-sub model where we wait for events like in the case of UMB messages.