Starting with OpenShift Container Platform 4.15, https://github.com/openshift/cluster-openshift-controller-manager-operator/blob/release-4.15/pkg/operator/internalimageregistry/cleanup_controller.go has been made available (as part of https://issues.redhat.com/browse/API-1642), causing all serviceAccount secret to get removed when the OpenShift Container Platform 4 - Image Registry is set to removed state and the secret is not in use. While this controller has good intention, it needs to be highlighted in the OpenShift Container Platform 4.15 - Release Notes as it's rather surprising to customers to see all secrets being removed after the OpenShift Container Platform 4.15 update completed. Not knowing why this is happening will trigger massive uncertainty and hence needs to be documented to help customers understand that this change is coming, why it's coming, the effect and overall impact.

OpenShift Container Platform 4.15

Always

1. Install OpenShift Container Platform 4.14
2. Set the Image Registry to Removed state
3. Update to OpenShift Container Platform 4.15

> $ oc get clusterversion
> NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
> version   4.14.25   True        False         2m26s   Cluster version is 4.14.25

> $ oc get config.imageregistry.operator.openshift.io/cluster -o json | jq .spec
> {
>   "httpSecret": "a3af443b3fe06f95322a[...]e821b22785825de8e39",
>   "logLevel": "Normal",
>   "managementState": "Removed",
>   "observedConfig": null,
>   "operatorLogLevel": "Normal",
>   "proxy": {},
>   "replicas": 2,
>   "requests": {
>     "read": {
>       "maxWaitInQueue": "0s"
>     },
>     "write": {
>       "maxWaitInQueue": "0s"
>     }
>   },
>   "rolloutStrategy": "RollingUpdate",
>   "storage": {
>     "managementState": "Managed",
>     "s3": {
>       "encrypt": true,
>       "region": "us-east-2",
>       "trustedCA": {
>         "name": ""
>       },
>       "virtualHostedStyle": false
>     }
>   },
>   "unsupportedConfigOverrides": null
> }

> $ oc get pod -n openshift-image-registry
> NAME                                               READY   STATUS    RESTARTS   AGE
> cluster-image-registry-operator-554cf4d4c5-6p68f   1/1     Running   0          24m
> node-ca-49p4b                                      1/1     Running   0          15m
> node-ca-cjl9q                                      1/1     Running   0          15m
> node-ca-gl88z                                      1/1     Running   0          15m
> node-ca-l4dvk                                      1/1     Running   0          15m
> node-ca-r484z                                      1/1     Running   0          15m
> node-ca-tvzxh                                      1/1     Running   0          15m

> $ oc get pod -n project-100
> NAME                                   READY   STATUS    RESTARTS   AGE
> sandbox-application-controller-0       1/1     Running   0          41s
> sandbox-redis-6d6cdb569b-lfp4k         1/1     Running   0          41s
> sandbox-repo-server-788d849479-tfvcn   1/1     Running   0          41s
> sandbox-server-77494b66b4-629hn        1/1     Running   0          41s

> $ oc get sa -n project-100
> NAME                                    SECRETS   AGE
> builder                                 1         75s
> default                                 1         75s
> deployer                                1         75s
> sandbox-argocd-application-controller   1         53s
> sandbox-argocd-grafana                  1         53s
> sandbox-argocd-redis                    1         53s
> sandbox-argocd-redis-ha                 1         53s
> sandbox-argocd-server                   1         53s

> $ oc get secret -n project-100
> NAME                                                    TYPE                                  DATA   AGE
> argocd-secret                                           Opaque                                5      58s
> builder-dockercfg-5qzmk                                 kubernetes.io/dockercfg               1      80s
> builder-token-7hcr4                                     kubernetes.io/service-account-token   4      80s
> default-dockercfg-rz6g7                                 kubernetes.io/dockercfg               1      80s
> default-token-589l8                                     kubernetes.io/service-account-token   4      80s
> deployer-dockercfg-5hsw6                                kubernetes.io/dockercfg               1      80s
> deployer-token-kgncp                                    kubernetes.io/service-account-token   4      80s
> sandbox-argocd-application-controller-dockercfg-82vpb   kubernetes.io/dockercfg               1      58s
> sandbox-argocd-application-controller-token-qhwjw       kubernetes.io/service-account-token   4      58s
> sandbox-argocd-grafana-dockercfg-z5sch                  kubernetes.io/dockercfg               1      58s
> sandbox-argocd-grafana-token-2cgcn                      kubernetes.io/service-account-token   4      58s
> sandbox-argocd-redis-dockercfg-wjg8r                    kubernetes.io/dockercfg               1      58s
> sandbox-argocd-redis-ha-dockercfg-qnqfw                 kubernetes.io/dockercfg               1      58s
> sandbox-argocd-redis-ha-token-2rhv2                     kubernetes.io/service-account-token   4      58s
> sandbox-argocd-redis-token-848nc                        kubernetes.io/service-account-token   4      58s
> sandbox-argocd-server-dockercfg-kdkl2                   kubernetes.io/dockercfg               1      58s
> sandbox-argocd-server-token-4p979                       kubernetes.io/service-account-token   4      58s
> sandbox-ca                                              kubernetes.io/tls                     3      58s
> sandbox-cluster                                         Opaque                                1      58s
> sandbox-default-cluster-config                          Opaque                                4      58s
> sandbox-tls                                             kubernetes.io/tls                     2      58s

---

> $ oc get clusterversion
> NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
> version   4.15.14   True        False         48s     Cluster version is 4.15.14
> 
> $ oc get config.imageregistry.operator.openshift.io/cluster -o json | jq .spec
> {
>   "httpSecret": "a3af443b3fe06f95322a[...]e821b22785825de8e39",
>   "logLevel": "Normal",
>   "managementState": "Removed",
>   "observedConfig": null,
>   "operatorLogLevel": "Normal",
>   "proxy": {},
>   "replicas": 2,
>   "requests": {
>     "read": {
>       "maxWaitInQueue": "0s"
>     },
>     "write": {
>       "maxWaitInQueue": "0s"
>     }
>   },
>   "rolloutStrategy": "RollingUpdate",
>   "storage": {
>     "managementState": "Managed",
>     "s3": {
>       "encrypt": true,
>       "region": "us-east-2",
>       "trustedCA": {
>         "name": ""
>       },
>       "virtualHostedStyle": false
>     }
>   },
>   "unsupportedConfigOverrides": null
> }

> $ oc get pod -n openshift-image-registry
> NAME                                               READY   STATUS    RESTARTS   AGE
> cluster-image-registry-operator-59486564cb-tms5b   1/1     Running   0          7m30s
> node-ca-4tlpt                                      1/1     Running   1          31m
> node-ca-6gpd9                                      1/1     Running   1          31m
> node-ca-h2wjg                                      1/1     Running   1          31m
> node-ca-q8lvx                                      1/1     Running   1          31m
> node-ca-qdrj6                                      1/1     Running   1          31m
> node-ca-smkjx                                      1/1     Running   1          31m

> $ oc get pod -n project-100
> NAME                                   READY   STATUS    RESTARTS   AGE
> sandbox-application-controller-0       1/1     Running   0          12m
> sandbox-redis-6d6cdb569b-4p26p         1/1     Running   0          15m
> sandbox-repo-server-788d849479-zl9cf   1/1     Running   0          15m
> sandbox-server-77494b66b4-dmtsd        1/1     Running   0          12m

> $ oc get sa -n project-100
> NAME                                    SECRETS   AGE
> builder                                 0         60m
> default                                 0         60m
> deployer                                0         60m
> sandbox-argocd-application-controller   0         60m
> sandbox-argocd-grafana                  0         60m
> sandbox-argocd-redis                    0         60m
> sandbox-argocd-redis-ha                 0         60m
> sandbox-argocd-server                   0         60m

> $ oc get secret -n project-100
> NAME                             TYPE                DATA   AGE
> argocd-secret                    Opaque              5      60m
> sandbox-ca                       kubernetes.io/tls   3      60m
> sandbox-cluster                  Opaque              1      60m
> sandbox-default-cluster-config   Opaque              4      60m
> sandbox-tls                      kubernetes.io/tls   2      60m

As we can see the number of secrets in namespace project-100 was very much reduced. This is unexpected yet desired behavior and hence must be documented in the OpenShift Container Platform 4.15 - Relase Notes to make customers aware.

It is also needed to fix the sentence: "After upgrading to 4.15, any existing service account token secrets are not deleted and continue to function" in the docs [1], as the "automatically generated" token secrets are deleted.

The warning "Do not rely on these automatically generated secrets for your own use; they might be removed in a future OpenShift Container Platform release." from 4.15 docs needs to be backported to 4.12+ docs.