-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
4.13
-
None
-
Moderate
-
No
-
CLOUD Sprint 248, CLOUD Sprint 249, CLOUD Sprint 250, CLOUD Sprint 251, CLOUD Sprint 252, CLOUD Sprint 253, CLOUD Sprint 254, CLOUD Sprint 255, CLOUD Sprint 256, CLOUD Sprint 257, CLOUD Sprint 258, CLOUD Sprint 259, CLOUD Sprint 260, CLOUD Sprint 261, CLOUD Sprint 263, CLOUD Sprint 264, CLOUD Sprint 262, CLOUD Sprint 265, CLOUD Sprint 266, CLOUD Sprint 267
-
20
-
Rejected
-
False
-
-
-
Bug Fix
-
Done
Description of problem:
Machine scale failed for GCP Marketplace cluster after upgrade from 4.12 to 4.13
Version-Release number of selected component (if applicable):
Upgrade from 4.12.26 to 4.13.0-0.nightly-2023-07-27-013427
How reproducible:
Always
Steps to Reproduce:
1.Install a 4.12 GCP Marketplace cluster
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.12.26 True False 24m Cluster version is 4.12.26
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME PHASE TYPE REGION ZONE AGE
huliu-41142-4cd9z-master-0 Running n2-standard-4 us-central1 us-central1-a 48m
huliu-41142-4cd9z-master-1 Running n2-standard-4 us-central1 us-central1-b 48m
huliu-41142-4cd9z-master-2 Running n2-standard-4 us-central1 us-central1-c 48m
huliu-41142-4cd9z-worker-a-z772h Running n2-standard-4 us-central1 us-central1-a 46m
huliu-41142-4cd9z-worker-b-7vb9n Running n2-standard-4 us-central1 us-central1-b 46m
2.Upgrade to 4.13
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.13.0-0.nightly-2023-07-27-013427 True False 15m Cluster version is 4.13.0-0.nightly-2023-07-27-013427
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME PHASE TYPE REGION ZONE AGE
huliu-41142-4cd9z-master-0 Running n2-standard-4 us-central1 us-central1-a 175m
huliu-41142-4cd9z-master-1 Running n2-standard-4 us-central1 us-central1-b 175m
huliu-41142-4cd9z-master-2 Running n2-standard-4 us-central1 us-central1-c 175m
huliu-41142-4cd9z-worker-a-z772h Running n2-standard-4 us-central1 us-central1-a 172m
huliu-41142-4cd9z-worker-b-7vb9n Running n2-standard-4 us-central1 us-central1-b 172m
3.Scale a machineset
liuhuali@Lius-MacBook-Pro huali-test % oc scale machineset huliu-41142-4cd9z-worker-a --replicas=2
machineset.machine.openshift.io/huliu-41142-4cd9z-worker-a scaled
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME PHASE TYPE REGION ZONE AGE
huliu-41142-4cd9z-master-0 Running n2-standard-4 us-central1 us-central1-a 5h35m
huliu-41142-4cd9z-master-1 Running n2-standard-4 us-central1 us-central1-b 5h35m
huliu-41142-4cd9z-master-2 Running n2-standard-4 us-central1 us-central1-c 5h35m
huliu-41142-4cd9z-worker-a-pdzg2 Failed 113s
huliu-41142-4cd9z-worker-a-z772h Running n2-standard-4 us-central1 us-central1-a 5h33m
huliu-41142-4cd9z-worker-b-7vb9n Running n2-standard-4 us-central1 us-central1-b 5h33m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-41142-4cd9z-worker-a-pdzg2 -oyaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
annotations:
machine.openshift.io/instance-state: Unknown
creationTimestamp: "2023-07-31T07:42:44Z"
finalizers:
- machine.machine.openshift.io
generateName: huliu-41142-4cd9z-worker-a-
generation: 1
labels:
machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: huliu-41142-4cd9z-worker-a
name: huliu-41142-4cd9z-worker-a-pdzg2
namespace: openshift-machine-api
ownerReferences:
- apiVersion: machine.openshift.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: MachineSet
name: huliu-41142-4cd9z-worker-a
uid: 43046eac-5ff5-4810-8e20-f0332128410f
resourceVersion: "163107"
uid: 1cd7d4d2-f231-457c-b21b-4ebc2d27363e
spec:
lifecycleHooks: {}
metadata: {}
providerSpec:
value:
apiVersion: machine.openshift.io/v1beta1
canIPForward: false
credentialsSecret:
name: gcp-cloud-credentials
deletionProtection: false
disks:
- autoDelete: true
boot: true
image: projects/redhat-marketplace-public/global/images/redhat-coreos-ocp-48-x86-64-202210040145
labels: null
sizeGb: 128
type: pd-ssd
kind: GCPMachineProviderSpec
machineType: n2-standard-4
metadata:
creationTimestamp: null
networkInterfaces:
- network: huliu-41142-4cd9z-network
subnetwork: huliu-41142-4cd9z-worker-subnet
projectID: openshift-qe
region: us-central1
serviceAccounts:
- email: huliu-41142-4cd9z-w@openshift-qe.iam.gserviceaccount.com
scopes:
- https://www.googleapis.com/auth/cloud-platform
shieldedInstanceConfig: {}
tags:
- huliu-41142-4cd9z-worker
userDataSecret:
name: worker-user-data
zone: us-central1-a
status:
conditions:
- lastTransitionTime: "2023-07-31T07:42:44Z"
status: "True"
type: Drainable
- lastTransitionTime: "2023-07-31T07:42:44Z"
message: Instance has not been created
reason: InstanceNotCreated
severity: Warning
status: "False"
type: InstanceExists
- lastTransitionTime: "2023-07-31T07:42:44Z"
status: "True"
type: Terminable
errorMessage: 'error launching instance: googleapi: Error 400: Invalid value for
field ''resource.shieldedInstanceConfig'': ''{ "enableVtpm": true, "enableIntegrityMonitoring":
true}''. Shielded VM Config can only be set when using a UEFI-compatible disk.,
invalid'
errorReason: InvalidConfiguration
lastUpdated: "2023-07-31T07:42:50Z"
phase: Failed
providerStatus:
conditions:
- lastTransitionTime: "2023-07-31T07:42:50Z"
message: 'googleapi: Error 400: Invalid value for field ''resource.shieldedInstanceConfig'':
''{ "enableVtpm": true, "enableIntegrityMonitoring": true}''. Shielded VM
Config can only be set when using a UEFI-compatible disk., invalid'
reason: MachineCreationFailed
status: "False"
type: MachineCreated
metadata: {}
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset huliu-41142-4cd9z-worker-a -oyaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
annotations:
machine.openshift.io/GPU: "0"
machine.openshift.io/memoryMb: "16384"
machine.openshift.io/vCPU: "4"
creationTimestamp: "2023-07-31T02:09:14Z"
generation: 2
labels:
machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z
name: huliu-41142-4cd9z-worker-a
namespace: openshift-machine-api
resourceVersion: "163067"
uid: 43046eac-5ff5-4810-8e20-f0332128410f
spec:
replicas: 2
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z
machine.openshift.io/cluster-api-machineset: huliu-41142-4cd9z-worker-a
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: huliu-41142-4cd9z-worker-a
spec:
lifecycleHooks: {}
metadata: {}
providerSpec:
value:
apiVersion: machine.openshift.io/v1beta1
canIPForward: false
credentialsSecret:
name: gcp-cloud-credentials
deletionProtection: false
disks:
- autoDelete: true
boot: true
image: projects/redhat-marketplace-public/global/images/redhat-coreos-ocp-48-x86-64-202210040145
labels: null
sizeGb: 128
type: pd-ssd
kind: GCPMachineProviderSpec
machineType: n2-standard-4
metadata:
creationTimestamp: null
networkInterfaces:
- network: huliu-41142-4cd9z-network
subnetwork: huliu-41142-4cd9z-worker-subnet
projectID: openshift-qe
region: us-central1
serviceAccounts:
- email: huliu-41142-4cd9z-w@openshift-qe.iam.gserviceaccount.com
scopes:
- https://www.googleapis.com/auth/cloud-platform
tags:
- huliu-41142-4cd9z-worker
userDataSecret:
name: worker-user-data
zone: us-central1-a
status:
availableReplicas: 1
fullyLabeledReplicas: 2
observedGeneration: 2
readyReplicas: 1
replicas: 2
Actual results:
Machine scale Failed
Expected results:
Machine should get Running, it shouldn’t validation when Shielded VM Config is not set.
Additional info:
Although we found this bug https://issues.redhat.com/browse/OCPBUGS-7367, but for the upgrade, the users didn’t set the parameter (shieldedInstanceConfig), didn’t want to use the feature either, but they cannot scale up the old machineset. That’s not convenient.
- blocks
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- links to
-
RHEA-2024:11038
OpenShift Container Platform 4.19.z bug fix update
[OCPBUGS-17079] Machine scale failed for GCP Marketplace cluster after upgrade from 4.12 to 4.13
Joel Speed
added a comment - rh-ee-nbrubake What do we need to move this bug forward? Joel Speed
added a comment - Discussed at planning, need to upload a 4.5 image to be able to test the UEFI settings from the older images
Joel Speed
added a comment - Discussed at planning, need to upload a 4.5 image to be able to test the UEFI settings from the older images Joel Speed
added a comment - rh-ee-nbrubake This one has been dormant for a while, anything we can do to move it along?
Joel Speed
added a comment - rh-ee-nbrubake This one has been dormant for a while, anything we can do to move it along? Joel Speed
added a comment - rh-ee-nbrubake I'd suggest poking the cloud credential operator team if you think special permissions are required. I believe for 4.5 we will still be using the non granular permissions in CI, but it's possible that we have changed over to using the granular permissions there as well. If that's the case, then adjusting those permissions may be a touch more complex
Joel Speed
added a comment - rh-ee-nbrubake I'd suggest poking the cloud credential operator team if you think special permissions are required. I believe for 4.5 we will still be using the non granular permissions in CI, but it's possible that we have changed over to using the granular permissions there as well. If that's the case, then adjusting those permissions may be a touch more complex I've tried a few different values for a cluster built with clusterbot, and I'm not coming up with a usable 4.5 image.
This same permission error pops up with projects/rhcos-cloud, projects/openshift-gce-devel, and projects/openshift-gce-devel-ci-2 (which is where my cluster was launched).
error launching instance: googleapi: Error 403: Required 'compute.images.useReadOnly' permission for 'projects/openshift-gce-devel-ci-2/global/images/rhcos-45-82-202008010929-0-gcp-x86-64', forbidden
Looking at it via the gcloud CLI:
❯ gcloud compute images describe projects/openshift-gce-devel/global/images/rhcos-45-82-202008010929-0-gcp-x86-64 ERROR: (gcloud.compute.images.describe) Could not fetch resource: - The resource 'projects/openshift-gce-devel/global/images/rhcos-45-82-202008010929-0-gcp-x86-64' was not found ❯ gcloud compute images describe projects/rhcos-cloud/global/images/rhcos-45-82-202008010929-0-gcp-x86-64 ERROR: (gcloud.compute.images.describe) Could not fetch resource: - Required 'compute.images.get' permission for 'projects/rhcos-cloud/global/images/rhcos-45-82-202008010929-0-gcp-x86-64'
I'm not sure where to go from here. Do we need to give special permissions to the service account within the cluster bot cluster? Copy the image into a project?
I dunno if it helps vs. GCP Marketplace, but 4.5 installers have:
"gcp": {
"image": "rhcos-45-82-202008010929-0-gcp-x86-64",
"url": "https://storage.googleapis.com/rhcos/rhcos/rhcos-45-82-202008010929-0-gcp-x86-64.tar.gz"
},
...
"gcp": {
"path": "rhcos-45.82.202008010929-0-gcp.x86_64.tar.gz",
"sha256": "025a54e4e53467c1c8aaa36b56eabe710a8a8a926c4c24c0650deb894cac2d4b",
"size": 896448321
},
You'll need to line that up with Ignition spec v2 user-data (see RFE-3001 discussing the 4.6 Ignition spec transition). See here for an example of me testing older AWS boot images on a modern cluster.
> Do you have a 4.5 release image that you could share? I cannot seem to find them still maintained on our normal registry
joelspeed The image I found from a problematic 4.5 cluster shows
ci-op-y4ilmr73-40cbb-gcs6z-rhcos-image
liuhuali@Lius-MacBook-Pro test1 % omg get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 11m Unable to apply 4.5.41: some cluster operators have not yet rolled out liuhuali@Lius-MacBook-Pro test1 % omg get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication Unknown Unknown True 40m cloud-credential 4.5.41 True True True 38m cluster-autoscaler 4.5.41 True False False 22m config-operator 4.5.41 True False False 22m console 4.5.41 Unknown True False 22m csi-snapshot-controller 4.5.41 True False False 37m dns 4.5.41 True False False 41m etcd 4.5.41 True False False 21m image-registry False True False 38m ingress False True True 37m insights 4.5.41 True False False 37m kube-apiserver 4.5.41 True False False 13m kube-controller-manager 4.5.41 True False False 19m kube-scheduler 4.5.41 True False False 17m kube-storage-version-migrator 4.5.41 False False False 42m machine-api 4.5.41 True False False 35m machine-approver 4.5.41 True False False 40m machine-config 4.5.41 True False False 21m marketplace 4.5.41 True False False 37m monitoring False True True 7m31s network 4.5.41 True False False 42m node-tuning 4.5.41 True False False 42m openshift-apiserver 4.5.41 True False False 19m openshift-controller-manager 4.5.41 True False False 36m openshift-samples 4.5.41 True False False 22m operator-lifecycle-manager 4.5.41 True False False 37m operator-lifecycle-manager-catalog 4.5.41 True False False 37m operator-lifecycle-manager-packageserver 4.5.41 True False False 38m service-ca 4.5.41 True False False 42m storage 4.5.41 True False False 38m liuhuali@Lius-MacBook-Pro test1 % omg get machine NAME PHASE TYPE REGION ZONE AGE ci-op-y4ilmr73-40cbb-gcs6z-master-0 44m ci-op-y4ilmr73-40cbb-gcs6z-master-1 44m ci-op-y4ilmr73-40cbb-gcs6z-master-2 44m ci-op-y4ilmr73-40cbb-gcs6z-worker-a-kt6dn 38m ci-op-y4ilmr73-40cbb-gcs6z-worker-b-272zb 38m ci-op-y4ilmr73-40cbb-gcs6z-worker-c-grxdz 38m liuhuali@Lius-MacBook-Pro test1 % omg get machine ci-op-y4ilmr73-40cbb-gcs6z-worker-c-grxdz -oyaml apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: creationTimestamp: '2024-02-21T02:38:02Z' finalizers: - machine.machine.openshift.io generateName: ci-op-y4ilmr73-40cbb-gcs6z-worker-c- generation: 1 labels: machine.openshift.io/cluster-api-cluster: ci-op-y4ilmr73-40cbb-gcs6z machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: ci-op-y4ilmr73-40cbb-gcs6z-worker-c managedFields: - apiVersion: machine.openshift.io/v1beta1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:machine.openshift.io/cluster-api-cluster: {} f:machine.openshift.io/cluster-api-machine-role: {} f:machine.openshift.io/cluster-api-machine-type: {} f:machine.openshift.io/cluster-api-machineset: {} f:ownerReferences: .: {} k:{"uid":"b240a1c5-92ed-4c98-9a7e-659186a7cc28"}: .: {} f:apiVersion: {} f:blockOwnerDeletion: {} f:controller: {} f:kind: {} f:name: {} f:uid: {} f:spec: .: {} f:metadata: {} f:providerSpec: .: {} f:value: .: {} f:apiVersion: {} f:canIPForward: {} f:credentialsSecret: {} f:deletionProtection: {} f:disks: {} f:kind: {} f:machineType: {} f:metadata: {} f:networkInterfaces: {} f:projectID: {} f:region: {} f:serviceAccounts: {} f:tags: {} f:userDataSecret: {} f:zone: {} manager: machineset-controller operation: Update time: '2024-02-21T02:38:02Z' - apiVersion: machine.openshift.io/v1beta1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: {} v:"machine.machine.openshift.io": {} f:status: {} manager: machine-controller-manager operation: Update time: '2024-02-21T02:38:05Z' name: ci-op-y4ilmr73-40cbb-gcs6z-worker-c-grxdz namespace: openshift-machine-api ownerReferences: - apiVersion: machine.openshift.io/v1beta1 blockOwnerDeletion: true controller: true kind: MachineSet name: ci-op-y4ilmr73-40cbb-gcs6z-worker-c uid: b240a1c5-92ed-4c98-9a7e-659186a7cc28 resourceVersion: '11190' selfLink: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/ci-op-y4ilmr73-40cbb-gcs6z-worker-c-grxdz uid: a0c227cf-6c3f-4af3-9f98-6faf26016065 spec: metadata: {} providerSpec: value: apiVersion: gcpprovider.openshift.io/v1beta1 canIPForward: false credentialsSecret: name: gcp-cloud-credentials deletionProtection: false disks: - autoDelete: true boot: true image: ci-op-y4ilmr73-40cbb-gcs6z-rhcos-image labels: null sizeGb: 128 type: pd-ssd kind: GCPMachineProviderSpec machineType: e2-standard-4 metadata: creationTimestamp: null networkInterfaces: - network: ci-op-y4ilmr73-40cbb-network subnetwork: ci-op-y4ilmr73-40cbb-worker-subnet projectID: XXXXXXXXXXXX region: us-central1 serviceAccounts: - email: ci-op-y4ilmr73-40cbb-gcs6z-w@XXXXXXXXXXXX.iam.gserviceaccount.com scopes: - https://www.googleapis.com/auth/cloud-platform tags: - ci-op-y4ilmr73-40cbb-gcs6z-worker userDataSecret: name: worker-user-data zone: us-central1-c liuhuali@Lius-MacBook-Pro test1 %
Joel Speed
added a comment - huliu@redhat.com Do you have a 4.5 release image that you could share? I cannot seem to find them still maintained on our normal registry
Joel Speed
added a comment - huliu@redhat.com Do you have a 4.5 release image that you could share? I cannot seem to find them still maintained on our normal registry > One more question - does QE do tests that upgrade from 4.2 to 4.13? If so, does this problem manifest?
Hi rh-ee-nbrubake I checked our prow CI, seems there is no job of upgrade from 4.2 to 4.13 on GCP, the earliest upgrade version on GCP is from 4.5, I will double check with the team, and if necessary, I will manually upgrade from 4.2 to 4.13 on GCP. But sorry, this may not start until after the holidays, because next week is the Spring Festival holiday in China, all Chinese associates are on holiday, and this upgrade path is so long that it cannot be completed in a short time.
By the way, the KCS looks good
rh-ee-miyadav already pre-merge verified this in another bug OCPBUGS-44671 so I think we are good to move forward.
By the way, the issue only occurs on the upgrade from 4.12 to 4.13, I will check the upgrade scenario again when we cherry-pick to 4.13.