Generate e2e tests for testing on openshift 12 with heightened security

As you know and as we shared before, the OCP releases 4.11 and 4.12 were impacted by some changes to align the Openshift authorization to the changes made in Kubernetes over Pod Security Admission. From 4.12 the global Pod Security enforcement will be changed from privileged to restricted but ideally, this should NOT affect the Operators distributed with OLM in the most common cases.

This email aims to let you know about the edge case scenarios which were identified and added under the section `Your solution might be affected in the following ways:`in the comprehensive guidance[1] provided. You do not need to read the whole guidance, and you can look for the subsection `Who is affected by these changes on 4.12 ?`

TL'DR: Upstream PSA changes may affect your operator on OCP 4.11 and 4.12 clusters, please review the comprehensive guidance document containing all details and caveats about the changes in both releases (4.11 and 4.12) and a huge FAQ section aimed toward Operator Authors.

For your convenience, we have included a summary of the common guidelines below which we believe are most helpful to operator authors.

🙋‍♀️What Operator Authors must keep in mind over this topic?

Ideally, if possible, ensure that your solutions can run as restricted[2]. The straightforward way to do it is via e2e tests where you will label the namespaces enforcing the restricted policy (pod-security.kubernetes.io/enforce:restricted)[3] and ensure that your Operator can work on them. You can check a code example in the SDK repository, see here[4].

Avoid creating Pod(s) directly, use Deployments instead (it might fail if you do not provide the security context config whole set to run as restricted)

Unless the consumers of your Operator need to enable cluster monitoring, avoid installing your Operator into namespaces prefixed with openshift. (the only exception is openshift-operators)

Do NOT set seccomProfile if you want that your workloads/Operator solution works on versions < 4.11. However, if you will distribute solutions only for 4.11+ then set it.

Do NOT set RunAsUser. It works on Kubernetes but will NOT work on OCP. In OCP, the SCC mutation will set a RunAsUser from the valid range of user IDs which is defined per namespace.

If you are setting this field because you are facing an error like `container has runAsNonRoot and image will run as root` then: you can sort it out by defining the USER ID into the container image (i.e. Dockerfile USER 1001:1001) which is the best recommendation. If you are unable to define the USER ID in the image then leave the RunAsUser and RunAsNonRoot empty. RunAsNonRoot as true will also be set via SCC mutation when the workload is admitted under SCC restricted-v2 policy.

However, If your solutions require escalated permissions:

Ensure the namespace containing your solution is labeled[7] accordingly so you are not dependent on the label syncer[8]. Assuming your operator already grants the appropriate SCC permissions to your operator service account, the label syncer should already be setting the PSA label on the namespace appropriately. Barring that, you can either update your operator to manage the namespace labels or include the namespace labeling as part of the manual install instructions. If you choose one of those options, you also MUST disable the label syncer[9] to ensure it does not reset the PSA label after your operator or user sets it.

Let the users of your Operator know why your solution requires raising permissions. You can add the information to the description of your Operator Bundle (CSV).

Workloads which require additional permissions must be managed by Service Accounts which have those additional permissions granted ( by default ONLY restricted-v2 permission is granted for all users ). You also might be able to configure RBAC permissions to grant access to use SCC[5] policies via the Service Account defined in the CSV. If you are using SDK you can, for example, use the markers[6], see:

//+kubebuilder:rbac:groups=security.openshift.io,resources=securitycontextconstraints,resourceNames=privileged,verbs=use

We hope that can help you out.

Cheers,

[1] - https://docs.google.com/document/d/1QLMkW6jdPNbnLima__6HLEtLqv93a0zC8DBkpldDrLY/edit#heading=h.omx9j9c49t49
[2] - https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted
[3] - https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-namespace-labels/
[4] - https://github.com/operator-framework/operator-sdk/tree/v1.23.0/testdata/go/v3/memcached-operator/test
[5] - https://docs.openshift.com/container-platform/4.11/authentication/managing-security-context-constraints.html
[6] -https://book.kubebuilder.io/reference/markers.html
[7] - https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-namespace-labels/
[8] - https://github.com/openshift/enhancements/blob/master/enhancements/authentication/pod-security-admission-autolabeling.md
[9] - https://docs.openshift.com/container-platform/4.11/authentication/understanding-and-managing-pod-security-admission.html#security-context-constraints-psa-opting_understanding-and-managing-pod-security-admission