Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-39039

Azure Encryption at Host Should be Independently Togglable from DiskEncryptionSetID

XMLWordPrintable

    • Important
    • None
    • Hypershift Sprint 259
    • 1
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required
    • In Progress

      Description of problem:

          EncryptionAtHost and DiskEncryptionSets are two features which should not be tightly coupled.  They should be able to be enabled / disabled independently.  Currently EncryptionAtHost is only enabled if DiskEncryptionSetID is a valid disk encryption set resource ID.


https://github.com/openshift/hypershift/blob/0cc82f7b102dcdf6e5d057255be1bdb1593d1203/hypershift-operator/controllers/nodepool/azure.go#L81-L88

      Version-Release number of selected component (if applicable):

      How reproducible:

          Every time

      Steps to Reproduce:

          1.See comments

      Actual results:

         EncryptionAtHost is only set if DiskEncryptionSetID is set.

      Expected results:

          EncryptionAtHost and DiskEncryptionSetID should be independently settable.

      Additional info:

          https://redhat-external.slack.com/archives/C075PHEFZKQ/p1724772123804009

            [OCPBUGS-39039] Azure Encryption at Host Should be Independently Togglable from DiskEncryptionSetID

            Feilian Xie (Inactive) added a comment -

            Verified with latest HO + 4.17 HC as well. 

            Feilian Xie (Inactive) added a comment - Verified with latest HO + 4.17 HC as well. 

            Feilian Xie (Inactive) added a comment - - edited

            Verified on 4.18. 

            Create HC without specifying encryption-at-host:
hypershift create cluster azure --fips --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --marketplace-offer=aro4 --marketplace-publisher=azureopenshift --marketplace-sku=aro_417 --marketplace-version=417.94.20240701 --diagnostics-storage-account-type=Managed --control-plane-availability-policy=HighlyAvailable --infra-availability-policy=HighlyAvailable --image-content-sources /tmp/secret/mgmt_icsp.yaml --name 1fb0e78578609be047da --node-pool-replicas 3 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain=qe1.azure.devcluster.openshift.com --pull-secret /etc/ci-pull-credentials/.dockerconfigjson --azure-creds /var/run/secrets/ci.openshift.io/cluster-profile/osServicePrincipal.json --location centralus --release-image registry.build03.ci.openshift.org/ci-op-fwvrsy0i/release@sha256:9f33c91217daf8fc0e3526498946ab0cc098ee91078cc86b90fec3c13aa6674a --encryption-key-id= --disk-encryption-set-id= --resource-group-name=ci-op-fwvrsy0i-8979e-rg --vnet-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-fwvrsy0i-8979e-vnet-rg/providers/Microsoft.Network/virtualNetworks/ci-op-fwvrsy0i-8979e-vnet --subnet-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-fwvrsy0i-8979e-vnet-rg/providers/Microsoft.Network/virtualNetworks/ci-op-fwvrsy0i-8979e-vnet/subnets/ci-op-fwvrsy0i-8979e-subnet --network-security-group-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-fwvrsy0i-8979e-nsg-rg/providers/Microsoft.Network/networkSecurityGroups/ci-op-fwvrsy0i-8979e-nsg

oc get hc -A
NAMESPACE   NAME                   VERSION                              KUBECONFIG                              PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
clusters    1fb0e78578609be047da   4.18.0-0.nightly-2024-09-11-221444   1fb0e78578609be047da-admin-kubeconfig   Completed   True        False         The hosted control plane is available

oc get np -n clusters    1fb0e78578609be047da -o jsonpath='{.spec.platform.azure.encryptionAtHost}'
Enabled

Check Azure console, VMs have encryption at host enabled. 

Create HC with --encryption-at-host=Disabled:
hypershift create cluster azure --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --network-type=Other --encryption-at-host=Disabled --image-content-sources /tmp/secret/mgmt_icsp.yaml --name 34ead5644f2231daf006 --node-pool-replicas 3 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain=qe1.azure.devcluster.openshift.com --pull-secret /etc/ci-pull-credentials/.dockerconfigjson --azure-creds /var/run/secrets/ci.openshift.io/cluster-profile/osServicePrincipal.json --location northcentralus --release-image registry.build03.ci.openshift.org/ci-op-lpc37237/release@sha256:9f33c91217daf8fc0e3526498946ab0cc098ee91078cc86b90fec3c13aa6674a --encryption-key-id= --disk-encryption-set-id= --resource-group-name= --vnet-id= --subnet-id= --network-security-group-id=

oc get hc -A
NAMESPACE   NAME                   VERSION                              KUBECONFIG                              PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
clusters    34ead5644f2231daf006   4.18.0-0.nightly-2024-09-11-221444   34ead5644f2231daf006-admin-kubeconfig   Completed   True        False         The hosted control plane is available

oc get np -n clusters    08abc3d493db75f18d63 -o jsonpath='{.spec.platform.azure.encryptionAtHost}'
Disabled

Check Azure console, VMs have encryption at host disabled. 

Create HC with --encryption-at-host=Enabled:
hypershift create cluster azure --fips --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --diagnostics-storage-account-type=UserManaged --diagnostics-storage-account-uri=https://d7e3f8e75734766sa.blob.core.windows.net/ --encryption-at-host=Enabled --image-content-sources /tmp/secret/mgmt_icsp.yaml --name d7e3f8e75734766003a5 --node-pool-replicas 3 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain=qe1.azure.devcluster.openshift.com --pull-secret /etc/ci-pull-credentials/.dockerconfigjson --azure-creds /var/run/secrets/ci.openshift.io/cluster-profile/osServicePrincipal.json --location southcentralus --release-image registry.build03.ci.openshift.org/ci-op-hmvtltz5/release@sha256:9f33c91217daf8fc0e3526498946ab0cc098ee91078cc86b90fec3c13aa6674a --encryption-key-id=https://ci-op-hmvtltz5-932e6-kv.vault.azure.net/keys/ci-op-hmvtltz5-932e6-key/55f01249ce2f438ea9ade5229ef3a57b --disk-encryption-set-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-hmvtltz5-932e6-rg/providers/Microsoft.Compute/diskEncryptionSets/ci-vtltz5-932e6-des-d --resource-group-name=ci-op-hmvtltz5-932e6-rg --vnet-id= --subnet-id= --network-security-group-id=

oc get hc -A
NAMESPACE   NAME                   VERSION                              KUBECONFIG                              PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
clusters    d7e3f8e75734766003a5   4.18.0-0.nightly-2024-09-11-221444   d7e3f8e75734766003a5-admin-kubeconfig   Completed   True        False         The hosted control plane is available

oc get np -n clusters d7e3f8e75734766003a5 -o jsonpath='{.spec.platform.azure.encryptionAtHost}'
Enabled

Check Azure console, VMs have encryption at host enabled. 

             

            Feilian Xie (Inactive) added a comment - - edited Verified on 4.18.  Create HC without specifying encryption-at-host: hypershift create cluster azure --fips --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --marketplace-offer=aro4 --marketplace-publisher=azureopenshift --marketplace-sku=aro_417 --marketplace-version=417.94.20240701 --diagnostics-storage-account-type=Managed --control-plane-availability-policy=HighlyAvailable --infra-availability-policy=HighlyAvailable --image-content-sources /tmp/secret/mgmt_icsp.yaml --name 1fb0e78578609be047da --node-pool-replicas 3 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain=qe1.azure.devcluster.openshift.com --pull-secret /etc/ci-pull-credentials/.dockerconfigjson --azure-creds / var /run/secrets/ci.openshift.io/cluster-profile/osServicePrincipal.json --location centralus --release-image registry.build03.ci.openshift.org/ci-op-fwvrsy0i/release@sha256:9f33c91217daf8fc0e3526498946ab0cc098ee91078cc86b90fec3c13aa6674a --encryption-key-id= --disk-encryption-set-id= --resource-group-name=ci-op-fwvrsy0i-8979e-rg --vnet-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-fwvrsy0i-8979e-vnet-rg/providers/Microsoft.Network/virtualNetworks/ci-op-fwvrsy0i-8979e-vnet --subnet-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-fwvrsy0i-8979e-vnet-rg/providers/Microsoft.Network/virtualNetworks/ci-op-fwvrsy0i-8979e-vnet/subnets/ci-op-fwvrsy0i-8979e-subnet --network-security-group-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-fwvrsy0i-8979e-nsg-rg/providers/Microsoft.Network/networkSecurityGroups/ci-op-fwvrsy0i-8979e-nsg oc get hc -A NAMESPACE   NAME                   VERSION                              KUBECONFIG                              PROGRESS    AVAILABLE   PROGRESSING   MESSAGE clusters    1fb0e78578609be047da   4.18.0-0.nightly-2024-09-11-221444   1fb0e78578609be047da-admin-kubeconfig   Completed   True        False         The hosted control plane is available oc get np -n clusters    1fb0e78578609be047da -o jsonpath= '{.spec.platform.azure.encryptionAtHost}' Enabled Check Azure console, VMs have encryption at host enabled.  Create HC with --encryption-at-host=Disabled: hypershift create cluster azure --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --network-type=Other --encryption-at-host=Disabled --image-content-sources /tmp/secret/mgmt_icsp.yaml --name 34ead5644f2231daf006 --node-pool-replicas 3 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain=qe1.azure.devcluster.openshift.com --pull-secret /etc/ci-pull-credentials/.dockerconfigjson --azure-creds / var /run/secrets/ci.openshift.io/cluster-profile/osServicePrincipal.json --location northcentralus --release-image registry.build03.ci.openshift.org/ci-op-lpc37237/release@sha256:9f33c91217daf8fc0e3526498946ab0cc098ee91078cc86b90fec3c13aa6674a --encryption-key-id= --disk-encryption-set-id= --resource-group-name= --vnet-id= --subnet-id= --network-security-group-id= oc get hc -A NAMESPACE   NAME                   VERSION                              KUBECONFIG                              PROGRESS    AVAILABLE   PROGRESSING   MESSAGE clusters    34ead5644f2231daf006   4.18.0-0.nightly-2024-09-11-221444   34ead5644f2231daf006-admin-kubeconfig   Completed   True        False         The hosted control plane is available oc get np -n clusters    08abc3d493db75f18d63 -o jsonpath= '{.spec.platform.azure.encryptionAtHost}' Disabled Check Azure console, VMs have encryption at host disabled.  Create HC with --encryption-at-host=Enabled: hypershift create cluster azure --fips --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --diagnostics-storage-account-type=UserManaged --diagnostics-storage-account-uri=https: //d7e3f8e75734766sa.blob.core.windows.net/ --encryption-at-host=Enabled --image-content-sources /tmp/secret/mgmt_icsp.yaml --name d7e3f8e75734766003a5 --node-pool-replicas 3 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain=qe1.azure.devcluster.openshift.com --pull-secret /etc/ci-pull-credentials/.dockerconfigjson --azure-creds / var /run/secrets/ci.openshift.io/cluster-profile/osServicePrincipal.json --location southcentralus --release-image registry.build03.ci.openshift.org/ci-op-hmvtltz5/release@sha256:9f33c91217daf8fc0e3526498946ab0cc098ee91078cc86b90fec3c13aa6674a --encryption-key-id=https://ci-op-hmvtltz5-932e6-kv.vault.azure.net/keys/ci-op-hmvtltz5-932e6-key/55f01249ce2f438ea9ade5229ef3a57b --disk-encryption-set-id=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-op-hmvtltz5-932e6-rg/providers/Microsoft.Compute/diskEncryptionSets/ci-vtltz5-932e6-des-d --resource-group-name=ci-op-hmvtltz5-932e6-rg --vnet-id= --subnet-id= --network-security-group-id= oc get hc -A NAMESPACE   NAME                   VERSION                              KUBECONFIG                              PROGRESS    AVAILABLE   PROGRESSING   MESSAGE clusters    d7e3f8e75734766003a5   4.18.0-0.nightly-2024-09-11-221444   d7e3f8e75734766003a5-admin-kubeconfig   Completed   True        False         The hosted control plane is available oc get np -n clusters d7e3f8e75734766003a5 -o jsonpath= '{.spec.platform.azure.encryptionAtHost}' Enabled Check Azure console, VMs have encryption at host enabled.   

            Feilian Xie (Inactive) added a comment -

            Also --enable-encryption-at-host Disabled sounds a little bit bizarre. We could make it --encryption-at-host Disabled 

            Feilian Xie (Inactive) added a comment - Also --enable-encryption-at-host Disabled sounds a little bit bizarre. We could make it --encryption-at-host Disabled 

            Feilian Xie (Inactive) added a comment -

            rh-ee-brcox 
            This function needs to be updated as well: 
            https://github.com/openshift/hypershift/blob/873c730433d2d435309222475c1e7b46f8eb4372/cmd/cluster/azure/create.go#L264

            Feilian Xie (Inactive) added a comment - rh-ee-brcox   This function needs to be updated as well:  https://github.com/openshift/hypershift/blob/873c730433d2d435309222475c1e7b46f8eb4372/cmd/cluster/azure/create.go#L264

            Feilian Xie (Inactive) added a comment -

            When I tried to render the manifests when creating HC with --enable-encryption-at-host Disabled:

            hypershift create cluster azure --fips --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --name fxie-hcp-2 --node-pool-replicas 2 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain qe1.azure.devcluster.openshift.com --pull-secret .dockerconfigjson --azure-creds $AZURE_CREDS --location $LOCATION --release-image registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2024-09-11-221444 --enable-encryption-at-host Disabled --render > hc2.yaml

            I got the following nodepool (encryptionAtHost field unspecified):

            apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  creationTimestamp: null
  name: fxie-hcp-2
  namespace: clusters
spec:
  arch: amd64
  clusterName: fxie-hcp-2
  management:
    autoRepair: false
    upgradeType: Replace
  nodeDrainTimeout: 0s
  nodeVolumeDetachTimeout: 0s
  platform:
    azure:
      diskSizeGB: 120
      image:
        azureImageType: ImageID
        imageID: /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/fxie-hcp-2-fxie-hcp-2-5m8tc/providers/Microsoft.Compute/images/rhcos.x86_64.vhd
      machineIdentityID: /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourcegroups/fxie-hcp-2-fxie-hcp-2-5m8tc/providers/Microsoft.ManagedIdentity/userAssignedIdentities/fxie-hcp-2-fxie-hcp-2-5m8tc
      subnetID: /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/fxie-hcp-2-fxie-hcp-2-5m8tc/providers/Microsoft.Network/virtualNetworks/fxie-hcp-2-fxie-hcp-2-5m8tc/subnets/default
      vmsize: Standard_D4s_v3
    type: Azure
  release:
    image: registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2024-09-11-221444
  replicas: 2
status:
  replicas: 0

            Feilian Xie (Inactive) added a comment - When I tried to render the manifests when creating HC with --enable-encryption-at-host Disabled: hypershift create cluster azure --fips --annotations=hypershift.openshift.io/pod-security-admission-label-override=baseline --name fxie-hcp-2 --node-pool-replicas 2 --base-domain qe.azure.devcluster.openshift.com --external-dns-domain qe1.azure.devcluster.openshift.com --pull-secret .dockerconfigjson --azure-creds $AZURE_CREDS --location $LOCATION --release-image registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2024-09-11-221444 --enable-encryption-at-host Disabled --render > hc2.yaml I got the following nodepool (encryptionAtHost field unspecified): apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool metadata:   creationTimestamp: null   name: fxie-hcp-2   namespace: clusters spec:   arch: amd64   clusterName: fxie-hcp-2   management:     autoRepair: false     upgradeType: Replace   nodeDrainTimeout: 0s   nodeVolumeDetachTimeout: 0s   platform:     azure:       diskSizeGB: 120       image:         azureImageType: ImageID         imageID: /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/fxie-hcp-2-fxie-hcp-2-5m8tc/providers/Microsoft.Compute/images/rhcos.x86_64.vhd       machineIdentityID: /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourcegroups/fxie-hcp-2-fxie-hcp-2-5m8tc/providers/Microsoft.ManagedIdentity/userAssignedIdentities/fxie-hcp-2-fxie-hcp-2-5m8tc       subnetID: /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/fxie-hcp-2-fxie-hcp-2-5m8tc/providers/Microsoft.Network/virtualNetworks/fxie-hcp-2-fxie-hcp-2-5m8tc/subnets/ default       vmsize: Standard_D4s_v3     type: Azure   release:     image: registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2024-09-11-221444   replicas: 2 status: replicas: 0

              rh-ee-brcox Bryan Cox
              bvesel@redhat.com Benjamin Vesel (Inactive)
              Feilian Xie Feilian Xie (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: