Last night a handful of hivei01ue1 clusters started alerting on high error rates in the CD and DNSZone controllers. The logs show these two controllers thrashing without backoff. The root cause seems to be that the Message field of the DNSError status condition on the DNSZone CR contains a RequestID, a per-request UUID from the AWS API. Since this changes every call, we're updating the CR every time, which triggers an immediate requeue.

Upon investigation, we already have a scrubber for this, and we're already using it in the appropriate spot. But its regex assumes a space between request and id, which is absent in the offending message.

This is possibly a result of the AWS SDK v2 transition (HIVE-2849); or possibly a recent AWS API change.