Uploaded image for project: 'Machine Config Operator'
  1. Machine Config Operator
  2. MCO-111

Actionable Error Messaging


    • Icon: Epic Epic
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • None
    • Actionable Error Messaging
    • False
    • False
    • To Do
    • OCPSTRAT-554 - Improving error handling, propagation, collection, and disambiguation for users
    • Impediment
    • OCPSTRAT-554Improving error handling, propagation, collection, and disambiguation for users
    • 67% To Do, 0% In Progress, 33% Done
    • 0
    • 0

      The error propagation is generally speaking not 1-to-1. The operator status will generally capture the pool status, but the full error from Controller/Daemon does not fully bubble up to pool/operator, and the journal logs with error generally don’t get bubbled up at all. This is very confusing for customers/admins working with the MCO without full understanding of the MCO’s internal mechanics:

      1. The real error is hard to find
      2. The error message is often generic and ambiguous
      3. The solution/workaround is not clear at all


      Using “unexpected on-disk state” as an example, this can be caused by any amount of the following:

      1. An incomplete update happened, and something rebooted the node
      2. The node upgrade was successful until rpm-ostree, which failed and atomically rolled back
      3. The user modified something manually
      4. Another operator modified something manually
      5. Some other service/network manager overwrote something MCO writes

      Etc. etc.


      Since error use cases are wide and varied, there are many improvements we can perform for each individual error state. This epic aims to propose targeted improvements to error messaging and propagation specifically. The goals being:


      1. De-ambigufying different error cases with the same message
      2. Adding more error catching, including journal logs and rpm-ostree errors
      3. Propagating full error messages further up the stack, up to the operator status in a clear manner
      4. Adding actionable fix/information messages alongside the error message


      With a side objective of observability, including reporting all the way to the operator status items such as:

      1. Reporting the status of all pools
      2. Pointing out current status of update/upgrade per pool
      3. What the update/upgrade is blocking on
      4. How to unblock the upgrade

      Approaches can include:

      1. Better error messaging starting with common error cases
      2. De-ambigufying config mismatch
      3. Capturing rpm-ostree logs from previous boot, in case of osimageurl mismatch errors
      4. Capturing full daemon error message back to pool/operator status
      5. Adding a new field to the MCO operator spec, that attempts to suggest fixes or where to look next, when an error occurs
      6. Adding better alerting messages for MCO errors

      • Options


        Docs Tracker Sub-task Closed Undefined Unassigned
        PX Tracker Sub-task Closed Undefined Unassigned
        QE Tracker Sub-task Closed Undefined Rio Liu
        TE Tracker Sub-task Closed Undefined Unassigned

            team-mco Team MCO
            mkrejci-1 Michelle Krejci
            Rio Liu Rio Liu
            0 Vote for this issue
            8 Start watching this issue