https://redhat-external.slack.com/archives/C075PHEFZKQ/p1727710473581569
https://docs.google.com/document/d/1xFJSXi71bl-fpAJBr2MM1iFdUqeQnlcneAjlH8ogQxQ/edit#heading=h.8e4x3inip35u

If we decided to drop the msi init and adapter and expose the certs in management cluster directly via Azure Key Vault Secret Store CSI Driver Pods volume. This would remove complexity and avoid the need for highly permissive pods with net access.

Action items:

• Each OpenShift component should authenticate via azidentity.NewDefaultAzureCredential
  and let it choose based on the exposed env variables. To account for cert rotation we might either leverage reloading in-process, or use the common OpenShift fsnotify +os.Exit().
  All the above could be implemented in a "library" like fashion that components ideally use as their only auth path. Otherwise they can for now keep their switch case based on the usecase for a gradual transition e.g. https://github.com/openshift/cluster-ingress-operator/compare/master...enxebre:cluster-ingress-operator:dev?expand=1

func azureCreds(options *azidentity.DefaultAzureCredentialOptions) (*azidentity.DefaultAzureCredential, error) {
if certPath := os.Getenv("AZURE_CLIENT_CERTIFICATE_PATH"); certPath != "" {
// Set up a watch on our config file; if it changes, we should exit -
// (we don't have the ability to dynamically reload config changes).
if err := watchForChanges(certPath, stopCh); err != nil

}

return azidentity.NewDefaultAzureCredential(options)
}

• For mocking besides production and getting CI passing:
  • Management cluster
    • Add the `--enable-addons azure-keyvault-secrets-provider` flag to the AZ CLI command that creates the AKS management cluster. This enables/installs the CSI secrets driver on the worker nodes.
    • Create an Azure Key Vault on the AKS cluster where the SP certs will be stored.
    • Create a managed identity that serve as the "user/thing" that reads the certs out of the Azure Key Vault for the CSI secret driver.
  • HyperShift Side
    • HyperShift CLI should provision a service principal to represent the MSI per HCP component and assigned the right roles/perms on them.
      • The Secret Store CSI driver/kube component should treat the cert as an MSI backing certificate. Regardless if you're using NewDefaultAzureCredential or NewClientCertificateCredential, it should work
    • The backing certificate for the SP should be stored in azure key vault in the same resource group as the AKS management cluster.
    • The HyperShift API will store:
      • The client ID and cert name of each SP for each HCP component
      • The Azure Key Vault name
      • The Azure Key Vault tenant ID
      • The client ID of the managed identity created to read the certs out of the Azure Key Vault
    • When the HCP deploys an OpenShift component that needs to authenticate with Azure:
      • HCP will supply the client ID for each SP 
      • HCP will add the volume mount for the CSI driver with the SP's cert name and the client ID of the managed identity that will read the secret from the Azure Key Vault

Engineering Notes:

Proof of Concept with Ingress as the example OpenShift component - https://github.com/openshift/hypershift/pull/4841/commits/35ac5fd3310b9199309e9e8a47ee661771ec71cf 

AZ CLI command to create the key vault

# Create Management Azure Key Vault
az keyvault create \
--name ${PREFIX} \
--resource-group ${AKS_RG} \
--location ${LOCATION} \
--enable-rbac-authorization 

AZ CLI command to create the managed identity for the key vault

## Create the managed identity for the Management Azure Key Vault
az identity create --name "${AZURE_KEY_VAULT_AUTHORIZED_USER}" --resource-group "${AKS_RG}"
AZURE_KEY_VAULT_AUTHORIZED_USER_ID=$(az identity show --name "${AZURE_KEY_VAULT_AUTHORIZED_USER}" --resource-group "${AKS_RG}" --query principalId --output tsv)
az role assignment create \
--assignee-object-id "${AZURE_KEY_VAULT_AUTHORIZED_USER_ID}" \
--role "Key Vault Secrets User" \
--scope /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/"${AKS_RG}" \
--assignee-principal-type ServicePrincipal 

AZ CLI command that creates a Service Principal with a backing cert stored in the Azure Key Vault

az ad sp create-for-rbac --name ingress --role "Contributor" --scopes /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${MANAGED_RG_NAME} --create-cert --cert ${CERTIFICATE_NAME} --keyvault ${KEY_VAULT_NAME}