Loading...

Type: Bug
Resolution: Unresolved
Priority: Blocker
Fix Version/s: OADP 1.5.0
Affects Version/s: OADP 1.50
Component/s: velero
Labels:
- oadp_419_req
- triaged

Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Fixed in Build:
oadp-operator-bundle-container-1.5.0-21
QEStatus:
ToDo
Intelligence Requested:
Market:

Cost of Delay:
0
WSJF:
0.000
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem:

I noticed issue while testing non admin backup and restore. Non admin user cannot exec into application pod after restore. We noticed that scc is getting changed to openshift.io/scc: privileged after restore from openshift.io/scc: restricted-v2.

slack thread;-
https://redhat-internal.slack.com/archives/C0144ECKUJ0/p1739282702636399

Attached error below:-

Error from server (Forbidden): pods "mysql-64c9d6466-l5rhd" is forbidden: exec operation is not allowed because the pod's security context exceeds your permissions: pods "mysql-64c9d6466-l5rhd" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000740000, 1000749999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

Version-Release number of selected component (if applicable):
OCP 4.19

How reproducible:
Always

Steps to Reproduce:
1. Create a non admin user and a namespace. Add namespace admin permission for the user.

2. Login to nonAdmin user and Use appm cli to deploy application in your namespace

$ appm deploy ocp-mysql -n <non-admin-user-namespace> -e '{"non_admin_user": "true"}'

3. Create filesystem backup of non admin namespace

4. Delete app namespace and trigger restore.

Actual results:

Namespace admin cannot exec into application after restore

Expected results:

Namespace admin should be able access the application after restore

Additional info:

Mysql pod yaml before backup.

$ oc get pod -n test-nac -o yaml mysql-64c9d6466-pxlf4 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.129.2.29/23"],"mac_address":"0a:58:0a:81:02:1d","gateway_ips":["10.129.2.1"],"routes":[{"dest":"10.128.0.0/14","nextHop":"10.129.2.1"},{"dest":"172.30.0.0/16","nextHop":"10.129.2.1"},{"dest":"169.254.0.5/32","nextHop":"10.129.2.1"},{"dest":"100.64.0.0/16","nextHop":"10.129.2.1"}],"ip_address":"10.129.2.29/23","gateway_ip":"10.129.2.1","role":"primary"}}'
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.129.2.29"
          ],
          "mac": "0a:58:0a:81:02:1d",
          "default": true,
          "dns": {}
      }]
    openshift.io/scc: restricted-v2
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
  creationTimestamp: "2025-02-11T15:05:45Z"
  generateName: mysql-64c9d6466-
  labels:
    app: mysql
    pod-template-hash: 64c9d6466
    testlabel: selectors
    testlabel2: foo
  name: mysql-64c9d6466-pxlf4
  namespace: test-nac
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: mysql-64c9d6466
    uid: 4c260e4c-d8d8-4f91-bf39-821d88fd2025
  resourceVersion: "188224"
  uid: 445f5fab-69ba-425e-b8c6-ee2c0c6df4db
spec:
  containers:
  - env:
    - name: MYSQL_USER
      valueFrom:
        secretKeyRef:
          key: database-user
          name: mysql
    - name: MYSQL_PASSWORD
      valueFrom:
        secretKeyRef:
          key: database-password
          name: mysql
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: database-root-password
          name: mysql
    - name: MYSQL_DATABASE
      valueFrom:
        secretKeyRef:
          key: database-name
          name: mysql
    image: quay.io/migqe/rhel8-mariadb:10.5
    imagePullPolicy: IfNotPresent
    name: mysql
    ports:
    - containerPort: 3306
      name: mysql
      protocol: TCP
    resources:
      limits:
        memory: 512Mi
      requests:
        memory: 512Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 1000780000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: mysql-data
    - mountPath: /test-data
      name: mysql-data1
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-dtnjg
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: default-dockercfg-htvf2
  nodeName: oadp-110351-qhbd8-worker-c-zk8x6
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000780000
    seLinuxOptions:
      level: s0:c28,c12
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  volumes:
  - name: mysql-data
    persistentVolumeClaim:
      claimName: mysql-data
  - name: mysql-data1
    persistentVolumeClaim:
      claimName: mysql-data1
  - name: kube-api-access-dtnjg
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
      - configMap:
          items:
          - key: service-ca.crt
            path: service-ca.crt
          name: openshift-service-ca.crt
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:06:02Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:05:49Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:06:02Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:06:02Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:05:49Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://39906228c80a47dc56f0da309d917917eb3eea935167f2ab9380d9b5a557e00f
    image: quay.io/migqe/rhel8-mariadb:10.5
    imageID: quay.io/migqe/rhel8-mariadb@sha256:c88f3c34c97f3739dcab17d015184c53834e668c32c072743e6e8308d22a0893
    lastState: {}
    name: mysql
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2025-02-11T15:06:02Z"
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: mysql-data
    - mountPath: /test-data
      name: mysql-data1
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-dtnjg
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 10.0.128.2
  hostIPs:
  - ip: 10.0.128.2
  phase: Running
  podIP: 10.129.2.29
  podIPs:
  - ip: 10.129.2.29
  qosClass: Burstable
  startTime: "2025-02-11T15:05:49Z"

Mysql pod yaml after restore

$ oc get pod mysql-64c9d6466-jjx7s -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.129.2.31/23"],"mac_address":"0a:58:0a:81:02:1f","gateway_ips":["10.129.2.1"],"routes":[{"dest":"10.128.0.0/14","nextHop":"10.129.2.1"},{"dest":"172.30.0.0/16","nextHop":"10.129.2.1"},{"dest":"169.254.0.5/32","nextHop":"10.129.2.1"},{"dest":"100.64.0.0/16","nextHop":"10.129.2.1"}],"ip_address":"10.129.2.31/23","gateway_ip":"10.129.2.1","role":"primary"}}'
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.129.2.31"
          ],
          "mac": "0a:58:0a:81:02:1f",
          "default": true,
          "dns": {}
      }]
    oadp.openshift.io/includes-dm-dc-fix: "true"
    openshift.io/backup-registry-hostname: image-registry.openshift-image-registry.svc:5000
    openshift.io/backup-server-version: "1.32"
    openshift.io/restore-registry-hostname: image-registry.openshift-image-registry.svc:5000
    openshift.io/restore-server-version: "1.32"
    openshift.io/scc: privileged
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
  creationTimestamp: "2025-02-11T15:29:39Z"
  labels:
    app: mysql
    pod-template-hash: 64c9d6466
    testlabel: selectors
    testlabel2: foo
    velero.io/backup-name: test-backup
    velero.io/restore-name: test-restore6
  name: mysql-64c9d6466-jjx7s
  namespace: test-nac
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: mysql-64c9d6466
    uid: 4e91d6fa-aa2b-4904-8dc6-62dd94d1db17
  resourceVersion: "196165"
  uid: 0afabb34-68b2-4646-84b5-b355c1d12fc6
spec:
  containers:
  - env:
    - name: MYSQL_USER
      valueFrom:
        secretKeyRef:
          key: database-user
          name: mysql
    - name: MYSQL_PASSWORD
      valueFrom:
        secretKeyRef:
          key: database-password
          name: mysql
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: database-root-password
          name: mysql
    - name: MYSQL_DATABASE
      valueFrom:
        secretKeyRef:
          key: database-name
          name: mysql
    image: quay.io/migqe/rhel8-mariadb:10.5
    imagePullPolicy: IfNotPresent
    name: mysql
    ports:
    - containerPort: 3306
      name: mysql
      protocol: TCP
    resources:
      limits:
        memory: 512Mi
      requests:
        memory: 512Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 1000780000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: mysql-data
    - mountPath: /test-data
      name: mysql-data1
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jttkt
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: default-dockercfg-2wv2f
  initContainers:
  - args:
    - 498186bf-8cbf-45ed-92fd-96faff861239
    command:
    - /velero-restore-helper
    env:
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: quay.io/konveyor/velero-restore-helper:latest
    imagePullPolicy: Always
    name: restore-wait
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
      requests:
        cpu: 100m
        memory: 128Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /restores/mysql-data
      name: mysql-data
    - mountPath: /restores/mysql-data1
      name: mysql-data1
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jttkt
      readOnly: true
  nodeName: oadp-110351-qhbd8-worker-c-zk8x6
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000780000
    seLinuxOptions:
      level: s0:c28,c12
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  volumes:
  - name: mysql-data
    persistentVolumeClaim:
      claimName: mysql-data
  - name: mysql-data1
    persistentVolumeClaim:
      claimName: mysql-data1
  - name: kube-api-access-jttkt
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
      - configMap:
          items:
          - key: service-ca.crt
            path: service-ca.crt
          name: openshift-service-ca.crt
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:30:00Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:30:08Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:30:09Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:30:09Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-02-11T15:29:43Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://590ace6ad1a0d587a67dee1fcf8fe75a334800b6235fc0253d1c4e1f7f73576b
    image: quay.io/migqe/rhel8-mariadb:10.5
    imageID: quay.io/migqe/rhel8-mariadb@sha256:c88f3c34c97f3739dcab17d015184c53834e668c32c072743e6e8308d22a0893
    lastState: {}
    name: mysql
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2025-02-11T15:30:08Z"
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: mysql-data
    - mountPath: /test-data
      name: mysql-data1
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jttkt
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 10.0.128.2
  hostIPs:
  - ip: 10.0.128.2
  initContainerStatuses:
  - containerID: cri-o://14d5cc1231e3ca52221dbb9bf95fa2a16252a26999eb9be040b55b231bc8bd4e
    image: quay.io/konveyor/velero-restore-helper:latest
    imageID: quay.io/konveyor/velero-restore-helper@sha256:87a73a1e5ee6b9f74f802c9ebceb269a4355c63fc1df813a282b4c2fc4fd0f97
    lastState: {}
    name: restore-wait
    ready: true
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: cri-o://14d5cc1231e3ca52221dbb9bf95fa2a16252a26999eb9be040b55b231bc8bd4e
        exitCode: 0
        finishedAt: "2025-02-11T15:30:08Z"
        reason: Completed
        startedAt: "2025-02-11T15:30:00Z"
    volumeMounts:
    - mountPath: /restores/mysql-data
      name: mysql-data
    - mountPath: /restores/mysql-data1
      name: mysql-data1
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jttkt
      readOnly: true
      recursiveReadOnly: Disabled
  phase: Running
  podIP: 10.129.2.31
  podIPs:
  - ip: 10.129.2.31
  qosClass: Burstable
  startTime: "2025-02-11T15:29:43Z"