Summary

On 12/14/2022, during final QE on MCE 2.1.3, the QE team observed that MCE 2.1.3 was already released and present in customer-facing Catalogs, although the MCE 2.1.3 Advisory was in a QE state and not SHIPPED LIVE. It was identified that this build matched the downstream RC3 build from 12/13/2022 as opposed to the current RC4 build which contained a last-minute patch. RC3 had already been cleared for release by QE, so no customers were effected or recieved non-release-quality code. Once we identified that RC3 was live, the Advisory was reverted to its RC3 state (from RC4) and re-pushed. Upon successful push, the Advisory entered a SHIPPED LIVE state. Concurrently, an MCE 2.1.4 was reversioned and rebuilt with only the changes originally included in MCE 2.1.3 RC4. MCE 2.1.4 was released on 12/15/2022 to include the fixes originally intended for 2.1.3 RC4.

Links

MCE 2.1.3 Advisory
MCE 2.1.4 Advisory

Timeline of Events

(Times in EST)

2022-12-12 20:41:55 UTC - 2.1.3 Advisory Moved to REL_PREP
2022-12-13 14:16:13 UTC - IN_PUSH after #acm-release signal to release
2022-12-13 17:07:48 UTC - PUSH FAILED
2022-12-13 17:10:48 UTC - IN_PUSH
2022-12-13 17:16:13 UTC - PUSH FAILED
2022-12-13 17:20:32 UTC - IN_PUSH
2022-12-13 18:00:58 UTC - PUSH FAILED
<intermittent state changes>
2022-12-13 19:31:51 UTC - Backed out to QE to rebuild with a last-minute fix, documented in Slack in thread
2022-12-14 10:18:43 EST - 2.1.3 is Discovered to be GA, documented in Slack
2022-12-14 11:52:18 EST - 2.1.3 GA is Discovered to be RC3 from previous failed pushes, documented in Slack
2022-12-14 12:16:57 EST - 2.1.3 Advisory has RC4 changes backed out, replaced by RC3, so it can be re-shipped Slack
2022-12-14 18:30:19 UTC - 2.1.3 Advisory moved to REL_PREP after Stage Push, Slack
2022-12-14 18:42:30 UTC - 2.1.3 Advisory IN_PUSH
2022-12-14 20:52:35 UTC - 2.1.3 Advisory SHIPPED_LIVE
2022-12-14 20:37:32 UTC - 2.1.4 Built
2022-12-14 21:36:50 UTC - 2.1.4 in QE
2022-12-15 01:06:06 UTC - 2.1.4 in REL_PREP
2022-12-15 14:43:13 UTC - 2.1.4 IN_PUSH
2022-12-15 20:57:35 UTC - SHIPPED_LIVE