There are permissions missing in the credentialsrequest for the cloud controller manager when attempting to exercise functionality in the CCM during a k8s service creation. 

The missing permissions are:
- Microsoft.Network/virtualNetworks/subnets/write
- Microsoft.Network/privatelinkservices/write
- Microsoft.Network/privatelinkservices/read
- Microsoft.Network/privatelinkservices/delete
- Microsoft.Network/loadBalancers/loadBalancingRules/read

    4.14.z+ 

    Every time

    1. Create an Azure workload identity cluster with custom roles or the built-in ARO roles
    2. Attempt to create a service of type load balancer with the below configuration.

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true" # Use an internal LB with PLS
    service.beta.kubernetes.io/azure-pls-create: "true"
spec:
  selector:
    app.kubernetes.io/name: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
  type: LoadBalancer
    

   You'll first hit the missing subnet permission, followed by missing privatelinkservices permission.

2s          Warning   SyncLoadBalancerFailed   service/my-service   Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '<client-id>' with object id '<object-id>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/write' over scope '/subscriptions/<subscription-id>/resourceGroups/bvesel/providers/Microsoft.Network/virtualNetworks/vnet/subnets/worker' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

If you grant permissions at that scope, you then get:

0s          Warning   SyncLoadBalancerFailed   service/my-service     Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '<client-id>' with object id '<object-id>' does not have authorization to perform action 'Microsoft.Network/privatelinkservices/write' over scope '/subscriptions/<subscription-id>/resourceGroups/aro-hj8744wi/providers/Microsoft.Network/privatelinkservices/pls-adb2fefe3d02d4d83a279061f1ba6099' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

    The service is able to create and configure the load balancer.  

    Each permission missing was an iteration of adding a permission to a custom role, recreating the service, and then seeing where CCM got tripped up.