Description of problem:

Statefulset failed to delete the volumes created after statefulset deletion.

Version-Release number of selected component (if applicable):

Payload: 4.11.0-0.nightly-2022-11-05-031756
version: v1.24.6+5157800

How reproducible:

Always 

Steps to Reproduce:

Tried sts volume deletion as per the doc: https://docs.openshift.com/container-platform/4.11/rest_api/workloads_apis/statefulset-apps-v1.html#spec-persistentvolumeclaimretentionpolicy

With enabled feature gate, 
oc get FeatureGate/cluster -o yaml
spec: 
  customNoUpgrade: 
    enabled: 
    - StatefulSetAutoDeletePVC 
  featureSet: CustomNoUpgrade 

After set of featuregate the kcm pods got rebooted and double verify for the feature gate parameter: --feature-gates=StatefulSetAutoDeletePVC=truekcm pod yaml file: oc -n openshift-kube-controller-manager get pod kube-controller-manager-ropatil-411gcp-2pb58-master-0.c.openshift-qe.internal  -o yaml | grep "feature-gates"feature-gates"        --requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 --tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt --tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key --allocate-node-cidrs=false --cert-dir=/var/run/kubernetes --cloud-config=/etc/kubernetes/static-pod-resources/configmaps/cloud-config/cloud.conf --cloud-provider=gce --cluster-cidr=10.128.0.0/14 --cluster-name=ropatil-411gcp-2pb58 --cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt --cluster-signing-duration=720h --cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key --configure-cloud-routes=false --controllers=* --controllers=-bootstrapsigner --controllers=-tokencleaner --controllers=-ttl --enable-dynamic-provisioning=true --feature-gates=StatefulSetAutoDeletePVC=true --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --kube-api-burst=300 --kube-api-qps=150 --leader-elect-resource-lock=configmapsleases --leader-elect-retry-period=3s --leader-elect=true --pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt --secure-port=10257 --service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key --service-cluster-ip-range=172.30.0.0/16 --use-service-account-credentials=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --tls-min-version=VersionTLS12

kubelet.conf
rohitpatil@ropatil-mac Downloads % oc debug node/ropatil-411gcp-2pb58-worker-a-5kbtq.c.openshift-qe.internal   "featureGates": {    "APIPriorityAndFairness": true,    "CSIMigrationAWS": false,    "CSIMigrationAzureFile": false,    "CSIMigrationGCE": false,    "CSIMigrationvSphere": false,    "DownwardAPIHugePages": true,    "PodSecurity": true,    "RotateKubeletServerCertificate": true,    "StatefulSetAutoDeletePVC": true  }, 

 
Create sts with parameters set,
sts.yaml
{
  "apiVersion": "apps/v1",
  "kind": "StatefulSet",
  "metadata":

,
  "spec": {
    "persistentVolumeClaimRetentionPolicy":

,
    "podManagementPolicy": "OrderedReady",
    "replicas": 2,
    "selector": {
      "matchLabels":

    },
    "template": {
      "metadata": {
        "labels":

,
        "annotations":

      },
      "spec": {
        "persistentVolumeClaimRetentionPolicy":

,
        "terminationGracePeriodSeconds": 0,
        "containers": [
          {
            "image": "quay.io/openshifttest/hello-openshift@sha256:4200f438cf2e9446f6bcff9d67ceea1f69ed07a2f83363b7fb52529f7ddd8a83",
            "imagePullPolicy": "IfNotPresent",
            "name": "mysts",
            "ports": [
             

            ],
            "resources": {},
            "securityContext": {
              "capabilities": {},
              "privileged": false
            },
            "terminationMessagePath": "/dev/termination-log",
            "volumeMounts": [
             

            ]
          }
        ],
        "dnsPolicy": "ClusterFirst",
        "restartPolicy": "Always",
        "serviceAccount": ""
      }
    },
    "volumeClaimTemplates": [
      {
        "metadata":

,
        "spec": {
          "storageClassName": "standard-csi",
          "volumeMode": "Filesystem",
          "accessModes": [
            "ReadWriteOnce"
          ],
          "resources": {
            "requests":

          }
        }
      }
    ]
  }
}
rohitpatil@ropatil-mac Downloads % oc get pvc,pod -n testropatil -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/sts-vol-mysts-0 Bound pvc-1cf6e73b-0e0b-47ed-ac69-91f2f9be10cc 2Gi RWO standard-csi 110s Filesystem
persistentvolumeclaim/sts-vol-mysts-1 Bound pvc-ca343845-5ef0-44a0-96b1-724074963609 2Gi RWO standard-csi 93s Filesystem

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/mysts-0 1/1 Running 0 110s 10.131.0.18 ropatil-411gcp-2pb58-worker-c-mdgnh.c.openshift-qe.internal <none> <none>
pod/mysts-1 1/1 Running 0 93s 10.129.2.18 ropatil-411gcp-2pb58-worker-b-2p7h8.c.openshift-qe.internal <none> <none>

rohitpatil@ropatil-mac Downloads % oc get sts -n testropatil
NAME READY AGE
mysts 2/2 100s
 
rohitpatil@ropatil-mac Downloads % oc get sts -n testropatil -o yaml
apiVersion: v1
items:

• apiVersion: apps/v1
  kind: StatefulSet
  metadata:
  creationTimestamp: "2022-11-07T06:55:06Z"
  generation: 1
  name: mysts
  namespace: testropatil
  spec:
  persistentVolumeClaimRetentionPolicy:
  whenDeleted: Delete
  whenScaled: Delete
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  selector:
  matchLabels:
  app: myapp
  serviceName: ""
  template:
  metadata:
  annotations:
  pod.alpha.kubernetes.io/initialized: "true"
  creationTimestamp: null
  labels:
  app: myapp
  spec:
  containers:
• image: quay.io/openshifttest/hello-openshift@sha256:4200f438cf2e9446f6bcff9d67ceea1f69ed07a2f83363b7fb52529f7ddd8a83
  imagePullPolicy: IfNotPresent
  name: mysts
  ports:
• containerPort: 8080
  protocol: TCP
  resources: {}
  securityContext:
  capabilities: {}
  privileged: false
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
• mountPath: /mnt/local
  name: sts-vol
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  terminationGracePeriodSeconds: 0
  updateStrategy:
  rollingUpdate:
  partition: 0
  type: RollingUpdate
  volumeClaimTemplates:
• apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
  creationTimestamp: null
  name: sts-vol
  spec:
  accessModes:
• ReadWriteOnce
  resources:
  requests:
  storage: 2Gi
  storageClassName: standard-csi
  volumeMode: Filesystem
  status:
  phase: Pending
  status:
  availableReplicas: 2
  collisionCount: 0
  currentReplicas: 2
  currentRevision: mysts-75db89494
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updateRevision: mysts-75db89494
  updatedReplicas: 2
  kind: List
  metadata:
  resourceVersion: ""
  rohitpatil@ropatil-mac Downloads % oc delete -f sts_fs.yaml
  statefulset.apps "mysts" deleted

rohitpatil@ropatil-mac Downloads % oc get pvc,pod -n testropatil -o wide
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
persistentvolumeclaim/sts-vol-mysts-0   Bound    pvc-1cf6e73b-0e0b-47ed-ac69-91f2f9be10cc   2Gi        RWO            standard-csi   15m   Filesystem
persistentvolumeclaim/sts-vol-mysts-1   Bound    pvc-ca343845-5ef0-44a0-96b1-724074963609   2Gi        RWO            standard-csi   15m   Filesystem

rohitpatil@ropatil-mac Downloads % oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
pvc-1cf6e73b-0e0b-47ed-ac69-91f2f9be10cc   2Gi        RWO            Delete           Bound    testropatil/sts-vol-mysts-0     standard-csi            15m
pvc-ca343845-5ef0-44a0-96b1-724074963609   2Gi        RWO            Delete           Bound    testropatil/sts-vol-mysts-1     standard-csi            15m

Additional info: 
Without featuregate enabled the sts output do not have the parameters. 

rohitpatil@ropatil-mac Downloads % oc get sts -n testropatil -o yaml
spec:
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10

kcm.logs(attaching the kcm logs) with Errors: 

"stateful_set.go:416] error syncing StatefulSet testropatil/mysts, requeuing: Could not update claim sts-vol-mysts-1 for delete policy ownerRefs: persistentvolumeclaims "sts-vol-mysts-1" is forbidden: cannot set an ownerRef on a resource you can't delete: , <nil>"

I1107 07:57:05.748231       1 operation_generator.go:398] AttachVolume.Attach succeeded for volume "pvc-ca343845-5ef0-44a0-96b1-724074963609" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/openshift-qe/zones/us-central1-b/disks/pvc-ca343845-5ef0-44a0-96b1-724074963609") from node "ropatil-411gcp-2pb58-worker-b-2p7h8.c.openshift-qe.internal" 
I1107 07:57:05.748342       1 event.go:294] "Event occurred" object="testropatil/mysts-1" fieldPath="" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"pvc-ca343845-5ef0-44a0-96b1-724074963609\" "
E1107 07:57:09.282007       1 stateful_set.go:416] error syncing StatefulSet testropatil/mysts, requeuing: Could not update claim sts-vol-mysts-1 for delete policy ownerRefs: persistentvolumeclaims "sts-vol-mysts-1" is forbidden: cannot set an ownerRef on a resource you can't delete: , <nil>
I1107 08:01:42.650299       1 operation_generator.go:513] DetachVolume.Detach succeeded for volume "pvc-ca343845-5ef0-44a0-96b1-724074963609" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/openshift-qe/zones/us-central1-b/disks/pvc-ca343845-5ef0-44a0-96b1-724074963609") on node "ropatil-411gcp-2pb58-worker-b-2p7h8.c.openshift-qe.internal"  

Actual results:

Volume is not getting deleted after statefulset is deleted

Expected results:

Volume should get deleted after statefulset is deleted