Uploaded image for project: 'Serverless logic'
  1. Serverless logic
  2. SRVLOGIC-700

[AGILE] Monitor and stabilize OSL midstream nightly builds

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • None
    • Agile
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Test and Release 1.37

      The main purpose of this task is to actively monitor, identify issues and coordinate the resolution of OSL midstream builds with two main goals in mind:

       

      1.- Identify issues earlier in the development cycle that might eventually impact also the productization process

      2.- Keep healthy nightly builds available to consume by third parties, like the RHDHO team during their development cycles for feature development or testing.

       

      In order to achieve that there are a few items to implement:

      1.- Midstream build nightly dashboard with metrics like, number of consecutive build failures or the last 7d and 30d success rate %. The following spreadsheet can be used as an initial proposal

      2.- Establish some SLA values like, for example, max. number of consecutive build failures = 3

      3.- Set up a group of 3-4 engineers (aka CI/CD/Build crew) that will act as a first layer of contact in case an SLA is not met. The group will coordinate itself in a dedicated Slack channel and will figure out who and how to collaborate in order to bring the build back to stable

      4.- Identify also a group lead which will be accountable for:

         a.- Actively monitoring the build system and keep the metrics in #1 up to date

         b.- When an SLA is not met, do an initial investigation 

         c.- Reach out to the CI Crew for issues resolution when required

      5.- Since everyone in the team is eligible to collaborate in having stable builds, the CI Crew lead might ask for help to other team members with the support of  the respective reporting manager, in case the CI crew is not able to fix an issue.

              drosabrno Daniel Rosa
              david.magallanes David Gutierrez
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: