-
Bug
-
Resolution: Done-Errata
-
Undefined
-
4.17
Description of problem:
Using payload built with https://github.com/openshift/installer/pull/8666/ so that master instances can be provisioned from gen2 image, which is required when configuring security type in install-config.
Enable TrustedLaunch security type in install-config:
==================
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
azure:
encryptionAtHost: true
settings:
securityType: TrustedLaunch
trustedLaunch:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
Launch capi-based installation, installer failed after waiting 15min for machines to provision...
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima08conf01-9vgq5-rg/providers/Microsoft.Compute/galleries/gallery_jima08conf01_9vgq5/images/jima08conf01-9vgq5
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima08conf01-9vgq5-rg/providers/Microsoft.Compute/galleries/gallery_jima08conf01_9vgq5/images/jima08conf01-9vgq5-gen2
INFO Created manifest *v1beta1.AzureMachine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-bootstrap
INFO Created manifest *v1beta1.AzureMachine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-master-0
INFO Created manifest *v1beta1.AzureMachine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-master-1
INFO Created manifest *v1beta1.AzureMachine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-master-2
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-bootstrap
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-master-0
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-master-1
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-master-2
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-bootstrap
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=jima08conf01-9vgq5-master
INFO Waiting up to 15m0s (until 6:26AM UTC) for machines [jima08conf01-9vgq5-bootstrap jima08conf01-9vgq5-master-0 jima08conf01-9vgq5-master-1 jima08conf01-9vgq5-master-2] to provision...
ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: control-plane machines were not provisioned within 15m0s: client rate limiter Wait returned an error: context deadline exceeded
INFO Shutting down local Cluster API control plane...
INFO Stopped controller: Cluster API
INFO Stopped controller: azure infrastructure provider
INFO Stopped controller: azureaso infrastructure provider
INFO Local Cluster API system has completed operations
In openshift-install.log,
time="2024-07-08T06:25:49Z" level=debug msg="\tfailed to reconcile AzureMachine: failed to reconcile AzureMachine service virtualmachine: failed to create or update resource jima08conf01-9vgq5-rg/jima08conf01-9vgq5-bootstrap (service: virtualmachine): PUT https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima08conf01-9vgq5-rg/providers/Microsoft.Compute/virtualMachines/jima08conf01-9vgq5-bootstrap"
time="2024-07-08T06:25:49Z" level=debug msg="\t--------------------------------------------------------------------------------"
time="2024-07-08T06:25:49Z" level=debug msg="\tRESPONSE 400: 400 Bad Request"
time="2024-07-08T06:25:49Z" level=debug msg="\tERROR CODE: BadRequest"
time="2024-07-08T06:25:49Z" level=debug msg="\t--------------------------------------------------------------------------------"
time="2024-07-08T06:25:49Z" level=debug msg="\t{"
time="2024-07-08T06:25:49Z" level=debug msg="\t \"error\": {"
time="2024-07-08T06:25:49Z" level=debug msg="\t \"code\": \"BadRequest\","
time="2024-07-08T06:25:49Z" level=debug msg="\t \"message\": \"Use of TrustedLaunch setting is not supported for the provided image. Please select Trusted Launch Supported Gen2 OS Image. For more information, see https://aka.ms/TrustedLaunch-FAQ.\""
time="2024-07-08T06:25:49Z" level=debug msg="\t }"
time="2024-07-08T06:25:49Z" level=debug msg="\t}"
time="2024-07-08T06:25:49Z" level=debug msg="\t--------------------------------------------------------------------------------"
time="2024-07-08T06:25:49Z" level=debug msg=" > controller=\"azuremachine\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AzureMachine\" AzureMachine=\"openshift-cluster-api-guests/jima08conf01-9vgq5-bootstrap\" namespace=\"openshift-cluster-api-guests\" name=\"jima08conf01-9vgq5-bootstrap\" reconcileID=\"bee8a459-c3c8-4295-ba4a-f3d560d6a68b\""
Looks like that capi-based installer missed to enable security features during creating gen2 image, which can be found in terraform code.
https://github.com/openshift/installer/blob/master/data/data/azure/vnet/main.tf#L166-L169
Gen2 image definition created by terraform:
$ az sig image-definition show --gallery-image-definition jima08conf02-4mrnz-gen2 -r gallery_jima08conf02_4mrnz -g jima08conf02-4mrnz-rg --query 'features'
[
{
"name": "SecurityType",
"value": "TrustedLaunch"
}
]
It's empty when querying from gen2 image created by using CAPI.
$ az sig image-definition show --gallery-image-definition jima08conf01-9vgq5-gen2 -r gallery_jima08conf01_9vgq5 -g jima08conf01-9vgq5-rg --query 'features'
$
Version-Release number of selected component (if applicable):
4.17 payload built from cluster-bot with PR https://github.com/openshift/installer/pull/8666/
How reproducible:
Always
Steps to Reproduce:
1. Enable security type in install-config
2. Create cluster by using CAPI
3.
Actual results:
Install failed.
Expected results:
Install succeeded.
Additional info:
It impacts installation with security type ConfidentialVM or TrustedLaunch enabled.
- blocks
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- links to
-
RHEA-2024:6122
OpenShift Container Platform 4.18.z bug fix update
(3 links to)
[OCPBUGS-36670] [CAPI Azure] Gen2 image definition missed security features enabled when configuring securitytype in install-config
Jinyun Ma
added a comment - Verified against 4.18.0-0.nightly-arm64-2024-10-23-004115, following scenarios are successful, move bug to VERIFIED.
1. security type trustedLaunched configured on both master and worker nodes
2. security type trustedLaunched configured on worker nodes only
3. security type confidentialVM + os disk security encryption type: VMGuestStateOnly configured on master and worker nodes
4. security type confidentialVM + os disk security encryption type: VMGuestStateOnly configured on worker nodes only
And installation with security type confidentialVM + os disk security encryption type:DiskWithVMGuestState is broken by bug OCPBUGS-18379, which need to be fixed on rhcos side.
Jinyun Ma
added a comment - Verified against 4.18.0-0.nightly-arm64-2024-10-23-004115, following scenarios are successful, move bug to VERIFIED.
1. security type trustedLaunched configured on both master and worker nodes
2. security type trustedLaunched configured on worker nodes only
3. security type confidentialVM + os disk security encryption type: VMGuestStateOnly configured on master and worker nodes
4. security type confidentialVM + os disk security encryption type: VMGuestStateOnly configured on worker nodes only
And installation with security type confidentialVM + os disk security encryption type:DiskWithVMGuestState is broken by bug OCPBUGS-18379 , which need to be fixed on rhcos side. Jinyun Ma
added a comment - Testing with PR https://github.com/openshift/installer/pull/9101/,
1. configured ConfidentailVM enabled + osDisk.securityEncryptionType: VMGuestStateOnly on both controlplane and compute, installation succeeded.
install-config: ========================= compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: azure: type: Standard_DC16ads_v5 settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: VMGuestStateOnly replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: azure: type: Standard_DC16ads_v5 settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: VMGuestStateOnly replicas: 3
2. I saw that the code change in PR also support to only enable ConfidentailVM enabled on compute, tested and installation succeeded as well.
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
azure:
type: Standard_DC16ads_v5
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform: {}
replicas: 3
3. configured ConfidentailVM enabled + osDisk.securityEncryptionType: DiskWithVMGuestState, got similar issues with https://issues.redhat.com/browse/OCPBUGS-18379, should be unrelated with PR installer#9101.
compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: azure: type: Standard_DC16ads_v5 settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: DiskWithVMGuestState replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: azure: type: Standard_DC16ads_v5 settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: DiskWithVMGuestState replicas: 3
sdasu@redhat.com Gen2 image definition security type is changed to "ConfidentialVmSupported", and the Gen2 image can create either Gen2 VM or ConfidentialVMs ( scenario 2). How about to update "TrustedLaunch" to "TrustedLaunchSupported" here together in the PR, so that scenario 2 can also be performed with TrustedLaunch security type?
Jinyun Ma
added a comment - Testing with PR https://github.com/openshift/installer/pull/9101/,
1. configured ConfidentailVM enabled + osDisk.securityEncryptionType: VMGuestStateOnly on both controlplane and compute, installation succeeded.
install-config:
=========================
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
azure:
type: Standard_DC16ads_v5
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
azure:
type: Standard_DC16ads_v5
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
replicas: 3
2. I saw that the code change in PR also support to only enable ConfidentailVM enabled on compute, tested and installation succeeded as well.
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
azure:
type: Standard_DC16ads_v5
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform: {}
replicas: 3
3. configured ConfidentailVM enabled + osDisk.securityEncryptionType: DiskWithVMGuestState, got similar issues with https://issues.redhat.com/browse/OCPBUGS-18379 , should be unrelated with PR installer#9101.
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
azure:
type: Standard_DC16ads_v5
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: DiskWithVMGuestState
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
azure:
type: Standard_DC16ads_v5
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: DiskWithVMGuestState
replicas: 3
sdasu@redhat.com Gen2 image definition security type is changed to "ConfidentialVmSupported", and the Gen2 image can create either Gen2 VM or ConfidentialVMs ( scenario 2). How about to update "TrustedLaunch" to "TrustedLaunchSupported" here together in the PR, so that scenario 2 can also be performed with TrustedLaunch security type? Jinyun Ma
added a comment - move back to Assigned because of my last comment . Jinyun Ma
added a comment - Verified against 4.18.0-0.nightly-2024-09-29-222603, get new error when installing with confidentialVM configured.
sdasu@redhat.com could you help to check? thanks.
install-config: ===================== compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: azure: type: Standard_DC16ads_v5 encryptionAtHost: true settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: VMGuestStateOnly replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: azure: type: Standard_DC16ads_v5 encryptionAtHost: true settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: VMGuestStateOnly replicas: 3 Error from output of installer: ================================= INFO Waiting up to 15m0s (until 6:12AM UTC) for network infrastructure to become ready... INFO Network infrastructure is ready INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconf01-z2rnv-rg/providers/Microsoft.Compute/galleries/gallery_jimaconf01_z2rnv/images/jimaconf01-z2rnv INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconf01-z2rnv-rg/providers/Microsoft.Compute/galleries/gallery_jimaconf01_z2rnv/images/jimaconf01-z2rnv-gen2 ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed provisioning resources after infrastructure ready: failed to create gallery image version 418.94.20240916: PUT https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconf01-z2rnv-rg/providers/Microsoft.Compute/galleries/gallery_jimaconf01_z2rnv/images/jimaconf01-z2rnv/versions/418.94.20240916 ERROR -------------------------------------------------------------------------------- ERROR RESPONSE 400: 400 Bad Request ERROR ERROR CODE: InvalidParameter ERROR -------------------------------------------------------------------------------- ERROR { ERROR "error": { ERROR "code": "InvalidParameter", ERROR "message": "Parameter 'galleryImageVersion.properties.storageProfile.osDiskImage.source.id' is not allowed.", ERROR "target": "galleryImageVersion.properties.storageProfile.osDiskImage.source.id" ERROR } ERROR } ERROR -------------------------------------------------------------------------------- ERROR INFO Shutting down local Cluster API controllers... INFO Stopped controller: Cluster API INFO Stopped controller: azure infrastructure provider INFO Stopped controller: azureaso infrastructure provider INFO Shutting down local Cluster API control plane... INFO Local Cluster API system has completed operations
Jinyun Ma
added a comment - Verified against 4.18.0-0.nightly-2024-09-29-222603, get new error when installing with confidentialVM configured.
sdasu@redhat.com could you help to check? thanks.
install-config:
=====================
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
azure:
type: Standard_DC16ads_v5
encryptionAtHost: true
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
azure:
type: Standard_DC16ads_v5
encryptionAtHost: true
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
replicas: 3
Error from output of installer:
=================================
INFO Waiting up to 15m0s (until 6:12AM UTC) for network infrastructure to become ready...
INFO Network infrastructure is ready
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconf01-z2rnv-rg/providers/Microsoft.Compute/galleries/gallery_jimaconf01_z2rnv/images/jimaconf01-z2rnv
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconf01-z2rnv-rg/providers/Microsoft.Compute/galleries/gallery_jimaconf01_z2rnv/images/jimaconf01-z2rnv-gen2
ERROR failed to fetch Cluster: failed to generate asset "Cluster" : failed to create cluster: failed provisioning resources after infrastructure ready: failed to create gallery image version 418.94.20240916: PUT https: //management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconf01-z2rnv-rg/providers/Microsoft.Compute/galleries/gallery_jimaconf01_z2rnv/images/jimaconf01-z2rnv/versions/418.94.20240916
ERROR --------------------------------------------------------------------------------
ERROR RESPONSE 400: 400 Bad Request
ERROR ERROR CODE: InvalidParameter
ERROR --------------------------------------------------------------------------------
ERROR {
ERROR "error" : {
ERROR "code" : "InvalidParameter" ,
ERROR "message" : "Parameter 'galleryImageVersion.properties.storageProfile.osDiskImage.source.id' is not allowed." ,
ERROR "target" : "galleryImageVersion.properties.storageProfile.osDiskImage.source.id"
ERROR }
ERROR }
ERROR --------------------------------------------------------------------------------
ERROR
INFO Shutting down local Cluster API controllers...
INFO Stopped controller: Cluster API
INFO Stopped controller: azure infrastructure provider
INFO Stopped controller: azureaso infrastructure provider
INFO Shutting down local Cluster API control plane...
INFO Local Cluster API system has completed operations
Jinyun Ma
added a comment - Verified against 4.18.0-0.nightly-2024-09-18-000823,
1. Gen2 image configured feature based on securityType setting in install-config, the issue stated in description is fixed.
2. Case 1, Case 2, osDisk setting in Case 3 from comments are all fixed.
3. Error about ConfidentialVM in Case 3 is not fixed yet, installer exited with the error when enabling ConfidentialVms.
INFO Waiting up to 15m0s (until 6:15AM UTC) for network infrastructure to become ready... INFO Network infrastructure is ready INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-lq78z-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_lq78z/images/jimaconfidential01-lq78z INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-lq78z-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_lq78z/images/jimaconfidential01-lq78z-gen2 ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed provisioning resources after infrastructure ready: failed to create gallery image version 417.94.20240827: PUT https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-lq78z-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_lq78z/images/jimaconfidential01-lq78z-gen2/versions/417.94.20240827 ERROR -------------------------------------------------------------------------------- ERROR RESPONSE 400: 400 Bad Request ERROR ERROR CODE: InvalidParameter ERROR -------------------------------------------------------------------------------- ERROR { ERROR "error": { ERROR "code": "InvalidParameter", ERROR "message": "Confidential VM is not supported for the source id ''. Currently Snapshot, Disk and VM sources are supported for the Confidential VM security type.", ERROR "target": "galleryImageVersion.properties.storageProfile.osDiskImage.source.id" ERROR } ERROR } ERROR -------------------------------------------------------------------------------- ERROR INFO Shutting down local Cluster API controllers... INFO Stopped controller: Cluster API INFO Stopped controller: azure infrastructure provider INFO Stopped controller: azureaso infrastructure provider INFO Shutting down local Cluster API control plane... INFO Local Cluster API system has completed operations
Discussed with sdasu@redhat.com offline last week, move bug to Assign to track item 3.
Jinyun Ma
added a comment - Verified against 4.18.0-0.nightly-2024-09-18-000823,
1. Gen2 image configured feature based on securityType setting in install-config, the issue stated in description is fixed.
2. Case 1, Case 2, osDisk setting in Case 3 from comments are all fixed.
3. Error about ConfidentialVM in Case 3 is not fixed yet, installer exited with the error when enabling ConfidentialVms.
INFO Waiting up to 15m0s (until 6:15AM UTC) for network infrastructure to become ready...
INFO Network infrastructure is ready
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-lq78z-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_lq78z/images/jimaconfidential01-lq78z
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-lq78z-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_lq78z/images/jimaconfidential01-lq78z-gen2
ERROR failed to fetch Cluster: failed to generate asset "Cluster" : failed to create cluster: failed provisioning resources after infrastructure ready: failed to create gallery image version 417.94.20240827: PUT https: //management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-lq78z-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_lq78z/images/jimaconfidential01-lq78z-gen2/versions/417.94.20240827
ERROR --------------------------------------------------------------------------------
ERROR RESPONSE 400: 400 Bad Request
ERROR ERROR CODE: InvalidParameter
ERROR --------------------------------------------------------------------------------
ERROR {
ERROR "error" : {
ERROR "code" : "InvalidParameter" ,
ERROR "message" : "Confidential VM is not supported for the source id ''. Currently Snapshot, Disk and VM sources are supported for the Confidential VM security type." ,
ERROR "target" : "galleryImageVersion.properties.storageProfile.osDiskImage.source.id"
ERROR }
ERROR }
ERROR --------------------------------------------------------------------------------
ERROR
INFO Shutting down local Cluster API controllers...
INFO Stopped controller: Cluster API
INFO Stopped controller: azure infrastructure provider
INFO Stopped controller: azureaso infrastructure provider
INFO Shutting down local Cluster API control plane...
INFO Local Cluster API system has completed operations
Discussed with sdasu@redhat.com offline last week, move bug to Assign to track item 3. Verified against 4.18.0-0.nightly-2024-09-11-154421, still hit some issues.
1. Configure platform.azure.settings.securityType under defaultMachinePlatform, panic error is not fixed.
install-config: ====================== platform: azure: baseDomainResourceGroupName: os4-common cloudName: AzurePublicCloud outboundType: Loadbalancer region: eastus defaultMachinePlatform: encryptionAtHost: true settings: securityType: TrustedLaunch trustedLaunch: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled Error in openshift-install.log when creating cluster. ====================== INFO Waiting up to 15m0s (until 2:17AM UTC) for network infrastructure to become ready... INFO Network infrastructure is ready INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima12trusted-l8bxh-rg/providers/Microsoft.Compute/galleries/gallery_jima12trusted_l8bxh/images/jima12trusted-l8bxh panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x4103e0c]goroutine 1 [running]: github.com/openshift/installer/pkg/infrastructure/azure.getControlPlaneSecurityType({{0x21c9aaa0, 0xc000c7fcb0}, 0xc00128d1c0, {0xc0014b9a70, 0x13}}) /go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:899 +0x2c github.com/openshift/installer/pkg/infrastructure/azure.(*Provider).InfraReady(0xc000f5e420, {0x21c7f9b0, 0xc000f54370}, {{0x21c9aaa0, 0xc000c7fcb0}, 0xc00128d1c0, {0xc0014b9a70, 0x13}}) /go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:439 +0x127b github.com/openshift/installer/pkg/infrastructure/clusterapi.(*InfraProvider).Provision(0xc0017c4e70, {0x21c7f9b0, 0xc000f54370}, {0xc000fcc020?, 0xc00128d040?}, 0xc000cece70) /go/src/github.com/openshift/installer/pkg/infrastructure/clusterapi/clusterapi.go:266 +0x17f3 github.com/openshift/installer/pkg/asset/cluster.(*Cluster).Generate(0x2638ada0, {0x21c7f9b0, 0xc000f54370}, 0xc000cece70) /go/src/github.com/openshift/installer/pkg/asset/cluster/cluster.go:141 +0x61d github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000f5b380, {0x21c7f9b0, 0xc000f54370}, {0x7f226bd17d20, 0x2638ada0}, {0x0, 0x0}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x6ec github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000f5b380, {0x21c7f9b0?, 0xc000f54370?}, {0x7f226bd17d20, 0x2638ada0}, {0x2634f800, 0x8, 0x8}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4e github.com/openshift/installer/pkg/asset/store.(*fetcher).FetchAndPersist(0xc0008da590, {0x21c7f9b0, 0xc000f54370}, {0x2634f800, 0x8, 0x8}) /go/src/github.com/openshift/installer/pkg/asset/store/assetsfetcher.go:47 +0x16b main.newCreateCmd.runTargetCmd.func3({0x7fff6a3dd531?, 0x4?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:306 +0x6a main.newCreateCmd.runTargetCmd.func4(0x2635a340, {0xc000f50c20?, 0x4?, 0x8241653?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:320 +0x102 github.com/spf13/cobra.(*Command).execute(0x2635a340, {0xc000f50c00, 0x2, 0x2}) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:989 +0xab1 github.com/spf13/cobra.(*Command).ExecuteC(0xc001254908) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1041 main.installerMain() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x3c6 main.main() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168
2. there should be no strong correlation between encryptionAtHost and settings.securityType in install-config, so it is valid case to only set securityType or set encryptionAtHost to false together in install-config, but in this case, feature is not set by installer, installation failed with same error as bug description.
install-config: ================= compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: azure: encryptionAtHost: false settings: securityType: TrustedLaunch trustedLaunch: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: azure: encryptionAtHost: false settings: securityType: TrustedLaunch trustedLaunch: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled replicas: 3 $ az sig image-definition show --gallery-image-definition jima12trusted02-22hz6-gen2 --gallery-name gallery_jima12trusted02_22hz6 -g jima12trusted02-22hz6-rg --query features $
3. Configured securityType to ConfidentialVM in install-config, property "feature" of gallery image definition was configured correctly this time by installer, but failed to create image versions. And I also found that platform.azure.osDisk.securityProfile configured in install-config is not written into capi machine manifests.
install-config: ======================= compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: azure: type: Standard_DC16ads_v5 encryptionAtHost: true settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: VMGuestStateOnly replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: azure: type: Standard_DC16ads_v5 encryptionAtHost: true settings: securityType: ConfidentialVM confidentialVM: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled osDisk: securityProfile: securityEncryptionType: VMGuestStateOnly Error in .openshift-install.log when creating cluster: ======================== INFO Waiting up to 15m0s (until 3:53AM UTC) for network infrastructure to become ready... INFO Network infrastructure is ready INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-tg9xf-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_tg9xf/images/jimaconfidential01-tg9xf INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-tg9xf-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_tg9xf/images/jimaconfidential01-tg9xf-gen2 ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed provisioning resources after infrastructure ready: failed to create gallery image version 418.94.20240905: PUT https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-tg9xf-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_tg9xf/images/jimaconfidential01-tg9xf-gen2/versions/418.94.20240905 ERROR -------------------------------------------------------------------------------- ERROR RESPONSE 400: 400 Bad Request ERROR ERROR CODE: InvalidParameter ERROR -------------------------------------------------------------------------------- ERROR { ERROR "error": { ERROR "code": "InvalidParameter", ERROR "message": "Confidential VM is not supported for the source id ''. Currently Snapshot, Disk and VM sources are supported for the Confidential VM security type.", ERROR "target": "galleryImageVersion.properties.storageProfile.osDiskImage.source.id" ERROR } ERROR } ERROR -------------------------------------------------------------------------------- ERROR INFO Shutting down local Cluster API controllers... INFO Stopped controller: Cluster API INFO Stopped controller: azure infrastructure provider INFO Stopped controller: azureaso infrastructure provider INFO Shutting down local Cluster API control plane... INFO Local Cluster API system has completed operations $ az sig image-definition show --gallery-image-definition jimaconfidential01-tg9xf-gen2 --gallery-name gallery_jimaconfidential01_tg9xf -g jimaconfidential01-tg9xf-rg --query features [ { "name": "SecurityType", "value": "ConfidentialVM" } ] <install-dir>/cluster-api/machines/10_inframachine_jimaconfidential01-j275l-bootstrap.yaml: ============================== osDisk: cachingType: ReadWrite diskSizeGB: 1024 managedDisk: storageAccountType: Premium_LRS osType: Linux securityProfile: encryptionAtHost: true securityType: ConfidentialVM uefiSettings: secureBootEnabled: true vTpmEnabled: true
sdasu@redhat.com could you help to check above issues? thanks. And FYI, feature enabling trusted launch and confidential VMs is in TP feature only now, which is documented in official install doc, so I think this issue might not be considered as release block?
Jinyun Ma
added a comment - - edited Verified against 4.18.0-0.nightly-2024-09-11-154421, still hit some issues.
1. Configure platform.azure.settings.securityType under defaultMachinePlatform, panic error is not fixed.
install-config:
======================
platform:
azure:
baseDomainResourceGroupName: os4-common
cloudName: AzurePublicCloud
outboundType: Loadbalancer
region: eastus
defaultMachinePlatform:
encryptionAtHost: true
settings:
securityType: TrustedLaunch
trustedLaunch:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
Error in openshift-install.log when creating cluster.
======================
INFO Waiting up to 15m0s (until 2:17AM UTC) for network infrastructure to become ready...
INFO Network infrastructure is ready
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima12trusted-l8bxh-rg/providers/Microsoft.Compute/galleries/gallery_jima12trusted_l8bxh/images/jima12trusted-l8bxh
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x4103e0c]goroutine 1 [running]:
github.com/openshift/installer/pkg/infrastructure/azure.getControlPlaneSecurityType({{0x21c9aaa0, 0xc000c7fcb0}, 0xc00128d1c0, {0xc0014b9a70, 0x13}})
/go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:899 +0x2c
github.com/openshift/installer/pkg/infrastructure/azure.(*Provider).InfraReady(0xc000f5e420, {0x21c7f9b0, 0xc000f54370}, {{0x21c9aaa0, 0xc000c7fcb0}, 0xc00128d1c0, {0xc0014b9a70, 0x13}})
/go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:439 +0x127b
github.com/openshift/installer/pkg/infrastructure/clusterapi.(*InfraProvider).Provision(0xc0017c4e70, {0x21c7f9b0, 0xc000f54370}, {0xc000fcc020?, 0xc00128d040?}, 0xc000cece70)
/go/src/github.com/openshift/installer/pkg/infrastructure/clusterapi/clusterapi.go:266 +0x17f3
github.com/openshift/installer/pkg/asset/cluster.(*Cluster).Generate(0x2638ada0, {0x21c7f9b0, 0xc000f54370}, 0xc000cece70)
/go/src/github.com/openshift/installer/pkg/asset/cluster/cluster.go:141 +0x61d
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000f5b380, {0x21c7f9b0, 0xc000f54370}, {0x7f226bd17d20, 0x2638ada0}, {0x0, 0x0})
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x6ec
github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000f5b380, {0x21c7f9b0?, 0xc000f54370?}, {0x7f226bd17d20, 0x2638ada0}, {0x2634f800, 0x8, 0x8})
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4e
github.com/openshift/installer/pkg/asset/store.(*fetcher).FetchAndPersist(0xc0008da590, {0x21c7f9b0, 0xc000f54370}, {0x2634f800, 0x8, 0x8})
/go/src/github.com/openshift/installer/pkg/asset/store/assetsfetcher.go:47 +0x16b
main.newCreateCmd.runTargetCmd.func3({0x7fff6a3dd531?, 0x4?})
/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:306 +0x6a
main.newCreateCmd.runTargetCmd.func4(0x2635a340, {0xc000f50c20?, 0x4?, 0x8241653?})
/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:320 +0x102
github.com/spf13/cobra.(*Command).execute(0x2635a340, {0xc000f50c00, 0x2, 0x2})
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:989 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0xc001254908)
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1041
main.installerMain()
/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x3c6
main.main()
/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168
2. there should be no strong correlation between encryptionAtHost and settings.securityType in install-config, so it is valid case to only set securityType or set encryptionAtHost to false together in install-config, but in this case, feature is not set by installer, installation failed with same error as bug description.
install-config:
=================
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
azure:
encryptionAtHost: false
settings:
securityType: TrustedLaunch
trustedLaunch:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
azure:
encryptionAtHost: false
settings:
securityType: TrustedLaunch
trustedLaunch:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
replicas: 3
$ az sig image-definition show --gallery-image-definition jima12trusted02-22hz6-gen2 --gallery-name gallery_jima12trusted02_22hz6 -g jima12trusted02-22hz6-rg --query features
$
3. Configured securityType to ConfidentialVM in install-config, property "feature" of gallery image definition was configured correctly this time by installer, but failed to create image versions. And I also found that platform.azure.osDisk.securityProfile configured in install-config is not written into capi machine manifests.
install-config:
=======================
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
azure:
type: Standard_DC16ads_v5
encryptionAtHost: true
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
azure:
type: Standard_DC16ads_v5
encryptionAtHost: true
settings:
securityType: ConfidentialVM
confidentialVM:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly
Error in .openshift-install.log when creating cluster:
========================
INFO Waiting up to 15m0s (until 3:53AM UTC) for network infrastructure to become ready...
INFO Network infrastructure is ready
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-tg9xf-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_tg9xf/images/jimaconfidential01-tg9xf
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-tg9xf-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_tg9xf/images/jimaconfidential01-tg9xf-gen2
ERROR failed to fetch Cluster: failed to generate asset "Cluster" : failed to create cluster: failed provisioning resources after infrastructure ready: failed to create gallery image version 418.94.20240905: PUT https: //management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jimaconfidential01-tg9xf-rg/providers/Microsoft.Compute/galleries/gallery_jimaconfidential01_tg9xf/images/jimaconfidential01-tg9xf-gen2/versions/418.94.20240905
ERROR --------------------------------------------------------------------------------
ERROR RESPONSE 400: 400 Bad Request
ERROR ERROR CODE: InvalidParameter
ERROR --------------------------------------------------------------------------------
ERROR {
ERROR "error" : {
ERROR "code" : "InvalidParameter" ,
ERROR "message" : "Confidential VM is not supported for the source id ''. Currently Snapshot, Disk and VM sources are supported for the Confidential VM security type." ,
ERROR "target" : "galleryImageVersion.properties.storageProfile.osDiskImage.source.id"
ERROR }
ERROR }
ERROR --------------------------------------------------------------------------------
ERROR
INFO Shutting down local Cluster API controllers...
INFO Stopped controller: Cluster API
INFO Stopped controller: azure infrastructure provider
INFO Stopped controller: azureaso infrastructure provider
INFO Shutting down local Cluster API control plane...
INFO Local Cluster API system has completed operations
$ az sig image-definition show --gallery-image-definition jimaconfidential01-tg9xf-gen2 --gallery-name gallery_jimaconfidential01_tg9xf -g jimaconfidential01-tg9xf-rg --query features
[
{
"name" : "SecurityType" ,
"value" : "ConfidentialVM"
}
]
<install-dir>/cluster-api/machines/10_inframachine_jimaconfidential01-j275l-bootstrap.yaml:
==============================
osDisk:
cachingType: ReadWrite
diskSizeGB: 1024
managedDisk:
storageAccountType: Premium_LRS
osType: Linux
securityProfile:
encryptionAtHost: true
securityType: ConfidentialVM
uefiSettings:
secureBootEnabled: true
vTpmEnabled: true
sdasu@redhat.com could you help to check above issues? thanks. And FYI, feature enabling trusted launch and confidential VMs is in TP feature only now, which is documented in official install doc, so I think this issue might not be considered as release block? 
OpenShift Jira Bot
added a comment - Hi sdasu@redhat.com,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
OpenShift Jira Bot
added a comment - Hi sdasu@redhat.com ,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified. Jinyun Ma
added a comment - Hi sdasu@redhat.com , pre-merge tested with installer PR#8967, it works well when setting securityType under controlPlane.platform.azure.settings.
But got panic error when setting under platform.azure.defaultMachinePlatform.settings.
install-config ================ platform: azure: baseDomainResourceGroupName: os4-common cloudName: AzurePublicCloud outboundType: Loadbalancer region: eastus defaultMachinePlatform: encryptionAtHost: true settings: securityType: TrustedLaunch trustedLaunch: uefiSettings: secureBoot: Enabled virtualizedTrustedPlatformModule: Enabled output for command "openshift-install create cluster" =================== $ ./openshift-install create cluster --dir ipi INFO Credentials loaded from file "/home/fedora/.azure/osServicePrincipal.json" INFO Consuming Install Config from target directory INFO Adding clusters... INFO Creating infrastructure resources... INFO Started local control plane with envtest INFO Stored kubeconfig for envtest in: /home/fedora/temp/4.18.0-0.test-2024-09-06-020928-ci-ln-647827b-latest/ipi/.clusterapi_output/envtest.kubeconfig INFO Running process: Cluster API with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:33501 --webhook-port=46645 --webhook-cert-dir=/tmp/envtest-serving-certs-3378449275 --kubeconfig=/home/fedora/temp/4.18.0-0.test-2024-09-06-020928-ci-ln-647827b-latest/ipi/.clusterapi_output/envtest.kubeconfig] INFO Running process: azure infrastructure provider with args [-v=2 --health-addr=127.0.0.1:36263 --webhook-port=43259 --webhook-cert-dir=/tmp/envtest-serving-certs-2360252394 --feature-gates=MachinePool=false --kubeconfig=/home/fedora/temp/4.18.0-0.test-2024-09-06-020928-ci-ln-647827b-latest/ipi/.clusterapi_output/envtest.kubeconfig] INFO Running process: azureaso infrastructure provider with args [-v=0 -metrics-addr=0 -health-addr=127.0.0.1:42135 -webhook-port=36591 -webhook-cert-dir=/tmp/envtest-serving-certs-3657438048 -crd-pattern= -crd-management=none] INFO Creating infra manifests... INFO Created manifest *v1.Namespace, namespace= name=openshift-cluster-api-guests INFO Created manifest *v1.Namespace, namespace= name=capz-system INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr-azure-client-secret INFO Created manifest *v1beta1.AzureClusterIdentity, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr INFO Created manifest *v1beta1.Cluster, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr INFO Created manifest *v1beta1.AzureCluster, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr INFO Done creating infra manifests INFO Creating kubeconfig entry for capi cluster jima06trusted-mtjbr INFO Waiting up to 15m0s (until 3:59AM UTC) for network infrastructure to become ready... INFO Network infrastructure is ready INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima06trusted-mtjbr-rg/providers/Microsoft.Compute/galleries/gallery_jima06trusted_mtjbr/images/jima06trusted-mtjbr panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x4102f79]goroutine 1 [running]: github.com/openshift/installer/pkg/infrastructure/azure.getControlPlaneSecurityType({{0x21c954e0, 0xc003c655f0}, 0xc002021ec0, {0xc001a5e7b0, 0x13}}) /go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:889 +0x39 github.com/openshift/installer/pkg/infrastructure/azure.(*Provider).InfraReady(0xc0011222c0, {0x21c7a390, 0xc001120000}, {{0x21c954e0, 0xc003c655f0}, 0xc002021ec0, {0xc001a5e7b0, 0x13}}) /go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:431 +0x127b github.com/openshift/installer/pkg/infrastructure/clusterapi.(*InfraProvider).Provision(0xc002059da0, {0x21c7a390, 0xc001120000}, {0xc00222a5c0?, 0xc002021d40?}, 0xc001ceb1d0) /go/src/github.com/openshift/installer/pkg/infrastructure/clusterapi/clusterapi.go:266 +0x17f3 github.com/openshift/installer/pkg/asset/cluster.(*Cluster).Generate(0x26384dc0, {0x21c7a390, 0xc001120000}, 0xc001ceb1d0) /go/src/github.com/openshift/installer/pkg/asset/cluster/cluster.go:141 +0x61d github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc00128aa20, {0x21c7a390, 0xc001120000}, {0x7f328d923fe0, 0x26384dc0}, {0x0, 0x0}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x6ec github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc00128aa20, {0x21c7a390?, 0xc001120000?}, {0x7f328d923fe0, 0x26384dc0}, {0x26349820, 0x8, 0x8}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4e github.com/openshift/installer/pkg/asset/store.(*fetcher).FetchAndPersist(0xc0005b46f0, {0x21c7a390, 0xc001120000}, {0x26349820, 0x8, 0x8}) /go/src/github.com/openshift/installer/pkg/asset/store/assetsfetcher.go:47 +0x16b main.newCreateCmd.runTargetCmd.func3({0x7ffda1d8e51e?, 0x3?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:306 +0x6a main.newCreateCmd.runTargetCmd.func4(0x26354360, {0xc001288240?, 0x4?, 0x82403d3?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:320 +0x102 github.com/spf13/cobra.(*Command).execute(0x26354360, {0xc001288220, 0x2, 0x2}) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:987 +0xab1 github.com/spf13/cobra.(*Command).ExecuteC(0xc00143cc08) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1039 main.installerMain() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x3c6 main.main() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168
Jinyun Ma
added a comment - Hi sdasu@redhat.com , pre-merge tested with installer PR#8967, it works well when setting securityType under controlPlane.platform.azure.settings.
But got panic error when setting under platform.azure.defaultMachinePlatform.settings.
install-config
================
platform:
azure:
baseDomainResourceGroupName: os4-common
cloudName: AzurePublicCloud
outboundType: Loadbalancer
region: eastus
defaultMachinePlatform:
encryptionAtHost: true
settings:
securityType: TrustedLaunch
trustedLaunch:
uefiSettings:
secureBoot: Enabled
virtualizedTrustedPlatformModule: Enabled
output for command "openshift-install create cluster"
===================
$ ./openshift-install create cluster --dir ipi
INFO Credentials loaded from file "/home/fedora/.azure/osServicePrincipal.json"
INFO Consuming Install Config from target directory
INFO Adding clusters...
INFO Creating infrastructure resources...
INFO Started local control plane with envtest
INFO Stored kubeconfig for envtest in: /home/fedora/temp/4.18.0-0.test-2024-09-06-020928-ci-ln-647827b-latest/ipi/.clusterapi_output/envtest.kubeconfig
INFO Running process: Cluster API with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:33501 --webhook-port=46645 --webhook-cert-dir=/tmp/envtest-serving-certs-3378449275 --kubeconfig=/home/fedora/temp/4.18.0-0.test-2024-09-06-020928-ci-ln-647827b-latest/ipi/.clusterapi_output/envtest.kubeconfig]
INFO Running process: azure infrastructure provider with args [-v=2 --health-addr=127.0.0.1:36263 --webhook-port=43259 --webhook-cert-dir=/tmp/envtest-serving-certs-2360252394 --feature-gates=MachinePool= false --kubeconfig=/home/fedora/temp/4.18.0-0.test-2024-09-06-020928-ci-ln-647827b-latest/ipi/.clusterapi_output/envtest.kubeconfig]
INFO Running process: azureaso infrastructure provider with args [-v=0 -metrics-addr=0 -health-addr=127.0.0.1:42135 -webhook-port=36591 -webhook-cert-dir=/tmp/envtest-serving-certs-3657438048 -crd-pattern= -crd-management=none]
INFO Creating infra manifests...
INFO Created manifest *v1.Namespace, namespace= name=openshift-cluster-api-guests
INFO Created manifest *v1.Namespace, namespace= name=capz-system
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr-azure-client-secret
INFO Created manifest *v1beta1.AzureClusterIdentity, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr
INFO Created manifest *v1beta1.Cluster, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr
INFO Created manifest *v1beta1.AzureCluster, namespace=openshift-cluster-api-guests name=jima06trusted-mtjbr
INFO Done creating infra manifests
INFO Creating kubeconfig entry for capi cluster jima06trusted-mtjbr
INFO Waiting up to 15m0s (until 3:59AM UTC) for network infrastructure to become ready...
INFO Network infrastructure is ready
INFO GalleryImage.ID=/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima06trusted-mtjbr-rg/providers/Microsoft.Compute/galleries/gallery_jima06trusted_mtjbr/images/jima06trusted-mtjbr
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x4102f79]goroutine 1 [running]:
github.com/openshift/installer/pkg/infrastructure/azure.getControlPlaneSecurityType({{0x21c954e0, 0xc003c655f0}, 0xc002021ec0, {0xc001a5e7b0, 0x13}})
/go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:889 +0x39
github.com/openshift/installer/pkg/infrastructure/azure.(*Provider).InfraReady(0xc0011222c0, {0x21c7a390, 0xc001120000}, {{0x21c954e0, 0xc003c655f0}, 0xc002021ec0, {0xc001a5e7b0, 0x13}})
/go/src/github.com/openshift/installer/pkg/infrastructure/azure/azure.go:431 +0x127b
github.com/openshift/installer/pkg/infrastructure/clusterapi.(*InfraProvider).Provision(0xc002059da0, {0x21c7a390, 0xc001120000}, {0xc00222a5c0?, 0xc002021d40?}, 0xc001ceb1d0)
/go/src/github.com/openshift/installer/pkg/infrastructure/clusterapi/clusterapi.go:266 +0x17f3
github.com/openshift/installer/pkg/asset/cluster.(*Cluster).Generate(0x26384dc0, {0x21c7a390, 0xc001120000}, 0xc001ceb1d0)
/go/src/github.com/openshift/installer/pkg/asset/cluster/cluster.go:141 +0x61d
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc00128aa20, {0x21c7a390, 0xc001120000}, {0x7f328d923fe0, 0x26384dc0}, {0x0, 0x0})
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x6ec
github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc00128aa20, {0x21c7a390?, 0xc001120000?}, {0x7f328d923fe0, 0x26384dc0}, {0x26349820, 0x8, 0x8})
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4e
github.com/openshift/installer/pkg/asset/store.(*fetcher).FetchAndPersist(0xc0005b46f0, {0x21c7a390, 0xc001120000}, {0x26349820, 0x8, 0x8})
/go/src/github.com/openshift/installer/pkg/asset/store/assetsfetcher.go:47 +0x16b
main.newCreateCmd.runTargetCmd.func3({0x7ffda1d8e51e?, 0x3?})
/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:306 +0x6a
main.newCreateCmd.runTargetCmd.func4(0x26354360, {0xc001288240?, 0x4?, 0x82403d3?})
/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:320 +0x102
github.com/spf13/cobra.(*Command).execute(0x26354360, {0xc001288220, 0x2, 0x2})
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:987 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0xc00143cc08)
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1039
main.installerMain()
/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x3c6
main.main()
/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Important: OpenShift Container Platform 4.18.1 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:6122