Details
-
Bug
-
Resolution: Done-Errata
-
Undefined
-
4.13
-
Moderate
-
No
-
False
-
-
Removed LatencySensitive featureset.
-
Removed Functionality
Description
Description of problem:
LatencySensitive was documented through 4.5 as a way to enable Topology Manager. But Topology Manager also went GA in 4.5. And in 4.6, LatencySensitive stopped being a trigger for Upgradeable=False 4.y-to-4.(y+1) update blocking. This combination of events leads to a few Telemetry-submitting clusters that are running modern 4.y but using the kind-of-forgotten-about LatencySensitive feature set. On top of this, recent openshift/api +openshift:enable:FeatureSets work has lead to some manifests being sharded between the Default and TechPreviewNoUpgrade feature sets, leaving those resources unmanaged for folks in the LatencySensitive set.
Version-Release number of selected component (if applicable):
The pivot happened from 4.12 to 4.13:
$ oc adm release extract --to 4.13 quay.io/openshift-release-dev/ocp-release:4.13.0-x86_64 $ grep -r 'release.openshift.io/feature-set:.*Default' 4.13 4.13/0000_10_config-operator_01_apiserver-Default.crd.yaml: release.openshift.io/feature-set: Default 4.13/0000_10_config-operator_01_infrastructure-Default.crd.yaml: release.openshift.io/feature-set: Default $ oc adm release extract --to 4.12 quay.io/openshift-release-dev/ocp-release:4.12.0-x86_64 $ grep -r 'release.openshift.io/feature-set:.*Default' 4.12 ...no hits...
How reproducible:
All the time, for LatencySensitive clusters.
Steps to Reproduce:
1. Launch a 4.12 cluster, e.g. with Cluster Bot launch 4.12.19 gcp.
2. Set the LatencySensitive feature set:
$ oc patch featuregate cluster --type=json --patch='[{"op":"add","path":"/spec/featureSet","value":"LatencySensitive"}]'
3. Set a channel that allows updates to 4.13: oc adm upgrade channel fast-4.13
4. Ack the Kube API removals:
$ oc -n openshift-config patch configmap admin-acks --type json --patch '[{"op":"add","path":"/data","value":{"ack-4.12-kube-1.26-api-removals-in-4.13":"true"}}]'
5. Update to 4.13: oc adm upgrade --to-latest
6. Wait for the update to complete, e.g. polling oc adm upgrade until it levels.
7. Check the changed-in-4.13 APIServer property:
$ oc get -o json customresourcedefinition apiservers.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.encryption.properties.type.enum'
8. Check a changed-in-4.13 Infrastructure property:
$ oc get -o json customresourcedefinition infrastructures.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.platformSpec.properties.type.enum[-2:]'
Actual results:
$ oc get -o json clusterversion version | jq -r '.status.history[] | .startedTime + " " + .completionTime + " " + .state + " " + .version' 2023-06-03T02:52:11Z 2023-06-03T04:20:57Z Completed 4.13.1 2023-06-03T02:18:46Z 2023-06-03T02:45:09Z Completed 4.12.19 $ oc get -o json customresourcedefinition apiservers.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.encryption.properties.type.enum' ["","identity","aescbc"] $ oc get -o json customresourcedefinition infrastructures.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.platformSpec.properties.type.enum[-2:]' ["AlibabaCloud","Nutanix"]
With 4.12's lack of aesgcm and External, despite the cluster claiming to have completed the update to 4.13.
The exclusion is also visible in the cluster-version operator logs:
$ oc -n openshift-cluster-version logs -l k8s-app=cluster-version-operator --tail -1 | grep -o 'excluding.*config-operator.*\(apiserver\|infrastructure\).*crd.*' | sort | uniq excluding 0000_10_config-operator_01_apiserver-Default.crd.yaml group=apiextensions.k8s.io kind=CustomResourceDefinition namespace= name=apiservers.config.openshift.io: "LatencySensitive" is required, and release.openshift.io/feature-set=Default excluding 0000_10_config-operator_01_apiserver-TechPreviewNoUpgrade.crd.yaml group=apiextensions.k8s.io kind=CustomResourceDefinition namespace= name=apiservers.config.openshift.io: "LatencySensitive" is required, and release.openshift.io/feature-set=TechPreviewNoUpgrade excluding 0000_10_config-operator_01_infrastructure-Default.crd.yaml group=apiextensions.k8s.io kind=CustomResourceDefinition namespace= name=infrastructures.config.openshift.io: "LatencySensitive" is required, and release.openshift.io/feature-set=Default excluding 0000_10_config-operator_01_infrastructure-TechPreviewNoUpgrade.crd.yaml group=apiextensions.k8s.io kind=CustomResourceDefinition namespace= name=infrastructures.config.openshift.io: "LatencySensitive" is required, and release.openshift.io/feature-set=TechPreviewNoUpgrade $ oc -n openshift-cluster-version logs -l k8s-app=cluster-version-operator --tail -1 | grep -o 'customresourcedefinition .*' | sort | uniq customresourcedefinition "alertmanagerconfigs.monitoring.coreos.com" (396 of 840) customresourcedefinition "alertmanagers.monitoring.coreos.com" (397 of 840) customresourcedefinition "authentications.config.openshift.io" (36 of 840) customresourcedefinition "authentications.operator.openshift.io" (275 of 840) customresourcedefinition "baremetalhosts.metal3.io" (224 of 840) ... customresourcedefinition "imagetagmirrorsets.config.openshift.io" (46 of 840) customresourcedefinition "imagetagmirrorsets.config.openshift.io" (46 of 840): CustomResourceDefinition imagetagmirrorsets.config.openshift.io does not declare an Established status condition: [] customresourcedefinition "ingresscontrollers.operator.openshift.io" (359 of 840) ...
Expected results:
$ oc get -o json customresourcedefinition apiservers.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.encryption.properties.type.enum' ["","identity","aescbc","aesgcm"] $ oc get -o json customresourcedefinition infrastructures.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.platformSpec.properties.type.enum[-2:]' ["Nutanix","External"]
With 4.13's expected aesgcm and External.
Additional info:
Assuming you do not have a controller that is attempting to manage FeatureGate's spec, you can recover your cluster by moving to the default feature set:
$ oc patch featuregate cluster --type=json --patch='[{"op":"remove","path":"/spec/featureSet"}]'
Shortly after which:
$ oc get -o json customresourcedefinition apiservers.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.encryption.properties.type.enum' ["","identity","aescbc","aesgcm"] $ oc get -o json customresourcedefinition infrastructures.config.openshift.io | jq -c '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties.platformSpec.properties.type.enum[-2:]' ["Nutanix","External"]
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-
- links to
-
RHEA-2023:5006
rpm
