Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54491

Azure CSI File Missing Permissions in CredentialsRequest

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • 4.20.0
    • 4.14.z
    • Storage / Operators
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • In Progress
    • Release Note Not Required
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          When attempting to exercise the azure csi file driver storage class private endpoint options, we encounter errors in a workload identity cluster with minimal permissions, meaning there are permissions from the credentials request. 
      
      Missing permissions are:
      - Microsoft.Network/privateDnsZones/read
      - Microsoft.Network/privateDnsZones/virtualNetworkLinks/write
      - Microsoft.Network/virtualNetworks/join/action
      - Microsoft.Network/privateEndpoints/write
      - Microsoft.Network/virtualNetworks/subnets/join/action
      - Microsoft.Storage/storageAccounts/PrivateEndpointConnectionsApproval/action
      - Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write

      Version-Release number of selected component (if applicable):

          4.14.z + 

      How reproducible:

          Every time

      Steps to Reproduce:

          1. Create an azure workload identity cluster with the current set of permissions outlined for azure file csi driver
          2. Create a private dns zone in your network resource group (defined in the azure cloud config) named: "privatelink.file.core.windows.net"
          3. Create a storage class with the driver option of "networkEndpointType" set to "privateEndpoint"
          4. Attempt to create a pvc and use it in a pod, it fails. 
      
      
      

      Actual results:

      {
        "error": {
          "code": "AuthorizationFailed",
          "message": "The client '<client-id>' with object id '<object-id>' does not have authorization to perform action 'Microsoft.Network/privateDnsZones/read' over scope '/subscriptions/<subscription-id>/resourceGroups/bvesel/providers/Microsoft.Network/privateDnsZones/privatelink.file.core.windows.net' or the scope is invalid. If access was recently granted, please refresh your credentials."
        }
      }
      
      Following resolution of that, the next one appears:
      
      {
        "error": {
          "code": "AuthorizationFailed",
          "message": "The client '<client-id>' with object id '<object-id>' does not have authorization to perform action 'Microsoft.Network/privateDnsZones/virtualNetworkLinks/write' over scope '/subscriptions/<subscription-id>/resourceGroups/bvesel/providers/Microsoft.Network/privateDnsZones/privatelink.file.core.windows.net/virtualNetworkLinks/vnet-vnetlink' or the scope is invalid. If access was recently granted, please refresh your credentials."
        }
      }
       
      
      Following resolution of that, you need
          "message": "The client 'dd91be29-1d24-4338-87bf-04073e1c1f23' with object id 'dd91be29-1d24-4338-87bf-04073e1c1f23' has permission to perform action 'Microsoft.Network/privateDnsZones/virtualNetworkLinks/write' on scope '/subscriptions/fe16a035-e540-4ab7-80d9-373fa9a3d6ae/resourceGroups/bvesel/providers/Microsoft.Network/privateDnsZones/privatelink.file.core.windows.net/virtualNetworkLinks/vnet-vnetlink'; however, it does not have permission to perform action(s) 'Microsoft.Network/virtualNetworks/join/action' on the linked scope(s) '/subscriptions/<subscription-id>/resourceGroups/bvesel/providers/Microsoft.Network/virtualNetworks/vnet' (respectively) or the linked scope(s) are invalid."
      
      Almost there, missing Private endpoint permissions
      E0402 02:23:15.862384       1 utils.go:82] GRPC error: rpc error: code = Internal desc = failed to ensure storage account: create private endpoint for storage account(f8cfaf76d6fe34f629b7ae9), resourceGroup(bvesel): Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client 'dd91be29-1d24-4338-87bf-04073e1c1f23' with object id 'dd91be29-1d24-4338-87bf-04073e1c1f23' does not have authorization to perform action 'Microsoft.Network/privateEndpoints/write' over scope '/subscriptions/<subscription-id>/resourceGroups/bvesel/providers/Microsoft.Network/privateEndpoints/f8cfaf76d6fe34f629b7ae9-pvtendpoint' or the scope is invalid. If access was recently granted, please refresh your credentials."}}
      
      
      Following resolution of that, the next one appears:
      E0402 02:36:08.811567       1 utils.go:82] GRPC error: rpc error: code = Internal desc = failed to ensure storage account: create private endpoint for storage account(f00597ddf81894301a8d571), resourceGroup(bvesel): Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"LinkedAuthorizationFailed","message":"The client 'dd91be29-1d24-4338-87bf-04073e1c1f23' with object id 'dd91be29-1d24-4338-87bf-04073e1c1f23' has permission to perform action 'Microsoft.Network/privateEndpoints/write' on scope '/subscriptions/<subscription>/resourceGroups/bvesel/providers/Microsoft.Network/privateEndpoints/f00597ddf81894301a8d571-pvtendpoint'; however, it does not have permission to perform action(s) 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked scope(s) '/subscriptions/<subscription>/resourceGroups/bvesel/providers/Microsoft.Network/virtualNetworks/vnet/subnets/worker' (respectively) or the linked scope(s) are invalid."}}
      
      
      Following resolution of that, the next one appears:
      E0402 02:43:28.895789       1 utils.go:82] GRPC error: rpc error: code = Internal desc = failed to ensure storage account: create private endpoint for storage account(fa9bb9d31287c4c668158cf), resourceGroup(bvesel): Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"LinkedAuthorizationFailed","message":"The client 'dd91be29-1d24-4338-87bf-04073e1c1f23' with object id 'dd91be29-1d24-4338-87bf-04073e1c1f23' has permission to perform action 'Microsoft.Network/privateEndpoints/write' on scope '/subscriptions/<subscription>/resourceGroups/bvesel/providers/Microsoft.Network/privateEndpoints/fa9bb9d31287c4c668158cf-pvtendpoint'; however, it does not have permission to perform action(s) 'Microsoft.Storage/storageAccounts/PrivateEndpointConnectionsApproval/action' on the linked scope(s) '/subscriptions/<subscription>/resourceGroups/aro-hj8744wi/providers/Microsoft.Storage/storageAccounts/fa9bb9d31287c4c668158cf' (respectively) or the linked scope(s) are invalid."}}
      
      After resolving the above...
      E0402 03:07:45.741799       1 utils.go:82] GRPC error: rpc error: code = Internal desc = failed to ensure storage account: create private DNS zone group - privateEndpoint(f029ab610e7904847b7fa00-pvtendpoint), vNetName(vnet), resourceGroup(bvesel): Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client 'dd91be29-1d24-4338-87bf-04073e1c1f23' with object id 'dd91be29-1d24-4338-87bf-04073e1c1f23' does not have authorization to perform action 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write' over scope '/subscriptions/<subscription>/resourceGroups/bvesel/providers/Microsoft.Network/privateEndpoints/f029ab610e7904847b7fa00-pvtendpoint/privateDnsZoneGroups/f029ab610e7904847b7fa00-dnszonegroup' or the scope is invalid. If access was recently granted, please refresh your credentials."}}
      
      
      And finally we're successful:
      $ oc get pvc
      NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      busybox-data-busybox-0   Bound    pvc-8cc621a1-af25-4331-9180-687c3551f9c6   10Gi       RWO            file           3m40s
      

      Expected results:

          No failures, PVC is able to be created and attached successfully.  

      Additional info:

      https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/driver-parameters.md    
      
      ---
      # storageclass.yaml
      allowVolumeExpansion: true
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
        name: file
      parameters:
        skuname: Premium_LRS
        networkEndpointType: privateEndpoint
      provisioner: file.csi.azure.com
      reclaimPolicy: Delete
      volumeBindingMode: WaitForFirstConsumer
      ---
      
      
      --- 
      # statefulset.yaml
      apiVersion: apps/v1
      kind: StatefulSet
      metadata:
        name: busybox
      spec:
        selector:
          matchLabels:
            app: busybox
        updateStrategy:
          type: RollingUpdate
        replicas: 1
        template:
          metadata:
            labels:
              app: busybox
          spec:
            containers:
            - name: busybox
              image: registry.hub.docker.com/library/busybox
              resources:
                limits:
                  memory: 2Gi
              volumeMounts:
                - name: busybox-data
                  mountPath: /usr/share/busybox
        volumeClaimTemplates:
        - metadata:
            name: busybox-data
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: "file"
            resources:
              requests:
                storage: 10Gi

              rbednar@redhat.com Roman Bednar
              bvesel.openshift Ben Vesel
              None
              None
              Wei Duan Wei Duan
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: