Since OCP 4.9, the chown command inside container does not work for system folders (/etc in this case)

4.9.latest

Easy

1. Build a container with below sample Dockerfile

FROM registry.redhat.io/ubi8/ubi
RUN groupadd -g 5000 test
RUN useradd -m -d /home/test -s /bin/bash -g test -u 5000 test
 
RUN chown test:test /etc/ && \
    chgrp test /etc/ && \
    chmod 755 /etc/

USER test
CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait" 

2. build the image
$podman build -t quay.io/rhn_support_xxx/etc-permission . 

$ podman images
REPOSITORY                                                                      TAG         IMAGE ID      CREATED        SIZE
quay.io/rhn_support_xxx/etc-permission                                         latest      2257355ed7b5  5 seconds ago  214 MB 

4. push the images to quay.io

$ podman login quay.io -u rhn_support_xxx -p xxx
$ podman push quay.io/rhn_support_xxx/etc-permission

5. login quay.io, change the repo visibility to public

6. create a new project, service account and optionally bind the most relaxed scc (privileged) to service account:

$ oc new-project test-etc-permission
$ oc create sa sa-common
$ oc adm policy add-scc-to-user privileged -z sa-common

7. create a deployment

$ cat test-etc-permission.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-etc-permission
  labels:
    app: test-etc-permission
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-etc-permission
  template:
    metadata:
      labels:
        app: test-etc-permission
    spec:
      containers:
      - name: test-pod
        image: quay.io/rhn_support_xxx/etc-permission
        args:
        - -c
        - sleep 1h
        command:
        - /bin/bash
      securityContext:
        fsGroup: 5000
        runAsGroup: 5000
        runAsUser: 5000
      serviceAccountName: sa-common

8. check the /etc/ folder permission inside the pod

Starting from OCP 4.9(tested version: 4.9.54), the same above steps does not be able to change the folder permission successfully, as below it was wrongly set to root:root

[root@upi-0 ~]# oc exec -it test-etc-permission-5d5f656494-sf8qx -- ls -ld /etc
drwxr-xr-x. 1 root root 6 Jan 19 05:52 /etc

On OCP 4.8.54, the /etc/ folder permission successfully got updated to test:test inside the pod:

[root@upi-0 ~]# oc exec -it test-etc-permission-5d5f656494-bhrp4 -- ls -ld /etc/
drwxr-xr-x. 1 test test 6 Jan 19 05:52 /etc/

The issue starts occur since 4.9. I tested 4.9/4.10/4.11, all the same behavior.