A recent PR removed the openshift.io/run-level label from the openshift-service-ca-operator namespace manifest, expecting it to remove the labels from namespaces when the CVO updated to a release with the trimmed manifest. But the CVO does not remove labels on updates to a release whose manifest lacks the label. Backing CVO code around that is years old, and only requires that all labels that exist in the manifest are matched in the in-cluster object. The CVO is agnostic about the presence of additional labels (and annotations) in the in-cluster object which do not appear in the committed manifest.

There was a lot of CVO churn back when that code was born, so there's not a lot of detail in commit messages explaining motivation, but I'd guess the idea was that cluster administrators might want to add their own annotations and labels to CVO-managed resources, and that the CVO didn't have a problem with them doing that. We've backed off of that position for some other properties, e.g. we now stomp out any un-manifested container ports. But possible avenues for this particular run-level case include:

a. Peeking into .metadata.managedFields for labels and annotations, and removing any from the in-cluster resource if they are not in the manifests and the manager is cluster-version-operator. This could be convenient, but it's also fairly magical, and I'd be worried that the magic would surprise someone else. Surprise is bad.

b. Having some non-CVO agent remove the label from the namespace. Could even be a human administrator, although that would be a lot of combined person-hours. I'm not sure if there's a convenient operator in place with creds to remove the label automatically.

c. Have the CVO require an exact match between manifest labels and annotations and the in-cluster objects. Might break folks who are somehow relying on this behavior today.

d. Teach the CVO about some denylist of labels and annotations which we stomp out of the in-cluster object unless they exist in the manifest (e.g. "nobody but the CVO is allowed to set openshift.io/... labels or annotations on CVO-managed objects".

e. See below.

f. See below.

This ticket is a place where we can kick around the costs and benefits of the different approaches, before making a sweeping change to CVO behavior.