-
Story
-
Resolution: Done
-
Blocker
-
None
-
None
-
None
-
Strategic Product Work
-
False
-
None
-
False
-
OCPSTRAT-153 - vSphere Zonal - Multiple datacenter, single vCenter support for vSphere IPI and UPI (GA)
As a openshift engineer I need to follow the process to move the api from tech preview to ga so it can be used by clusters not installed with TechPreviewNoUpgrade.
more to follow...
- links to
[SPLAT-820] [vsphere][zonal] Move tech preview api to GA
api#1388 has merged, just bumped cco#277 to pick up the changes and waiting on tests. will update installer#6782 in the mean time.
found an issue with a missing yaml-patch. that has been resolved and will ask QE to take another look.
performed testing with a build containing https://github.com/openshift/machine-config-operator/pull/3484 and removed the hold.
we are blocked waiting for https://github.com/openshift/machine-config-operator/pull/3484 to merge. enabling testing of nutanix e2e with https://github.com/openshift/release/pull/35658
confirmed that the cloud config is being converted to yaml as expected.
Taken from a system which was installed with no failure domains and had failure domains configured post-install.
$ oc get cm -n openshift-cloud-controller-manager cloud-conf -o yaml
apiVersion: v1
data:
cloud.conf: |
global:
insecureFlag: true
secretName: vsphere-creds
secretNamespace: kube-system
vcenter:
vcenter.vanderlab.net:
server: vcenter.vanderlab.net
port: 443
datacenters:
- vanderlab
labels:
zone: openshift-zone
region: openshift-region
kind: ConfigMap
metadata:
creationTimestamp: "2023-01-23T18:51:46Z"
name: cloud-conf
namespace: openshift-cloud-controller-manager
resourceVersion: "58637"
uid: fdbdc570-86a3-40ea-b0bd-4b70c98b55d8
confirmed that the cloud-config is generated with release image built from https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.13.0-0.ci/release/4.13.0-0.ci-2023-01-23-071032 and https://github.com/openshift/cluster-config-operator/pull/277 which is built against https://github.com/openshift/api/pull/1388 . installation was performed without the tech preview feature gate.
topology labels
$ oc get nodes -o=json | jq '.items[].metadata | "\(.name) \(.labels | to_entries | .[] | select(.key=="topology.kubernetes.io/region")) \(.labels | to_entries | .[] | select(.key=="topology.kubernetes.io/zone")) \(.labels | to_entries | .[] | select(.key=="topology.csi.vmware.com/openshift-region")) \(.labels | to_entries | .[] | select(.key=="topology.csi.vmware.com/openshift-zone"))"' "rvanderp5-dev-rmlzp-master-0 {\"key\":\"topology.kubernetes.io/region\",\"value\":\"vanderlab\"} {\"key\":\"topology.kubernetes.io/zone\",\"value\":\"zone-a\"} {\"key\":\"topology.csi.vmware.com/openshift-region\",\"value\":\"vanderlab\"} {\"key\":\"topology.csi.vmware.com/openshift-zone\",\"value\":\"zone-a\"}" "rvanderp5-dev-rmlzp-master-1 {\"key\":\"topology.kubernetes.io/region\",\"value\":\"vanderlab\"} {\"key\":\"topology.kubernetes.io/zone\",\"value\":\"zone-b\"} {\"key\":\"topology.csi.vmware.com/openshift-region\",\"value\":\"vanderlab\"} {\"key\":\"topology.csi.vmware.com/openshift-zone\",\"value\":\"zone-b\"}" "rvanderp5-dev-rmlzp-master-2 {\"key\":\"topology.kubernetes.io/region\",\"value\":\"vanderlab\"} {\"key\":\"topology.kubernetes.io/zone\",\"value\":\"zone-c\"} {\"key\":\"topology.csi.vmware.com/openshift-region\",\"value\":\"vanderlab\"} {\"key\":\"topology.csi.vmware.com/openshift-zone\",\"value\":\"zone-c\"}" "rvanderp5-dev-rmlzp-worker-0-tdw22 {\"key\":\"topology.kubernetes.io/region\",\"value\":\"vanderlab\"} {\"key\":\"topology.kubernetes.io/zone\",\"value\":\"zone-a\"} {\"key\":\"topology.csi.vmware.com/openshift-region\",\"value\":\"vanderlab\"} {\"key\":\"topology.csi.vmware.com/openshift-zone\",\"value\":\"zone-a\"}" "rvanderp5-dev-rmlzp-worker-1-ppplr {\"key\":\"topology.kubernetes.io/region\",\"value\":\"vanderlab\"} {\"key\":\"topology.kubernetes.io/zone\",\"value\":\"zone-b\"} {\"key\":\"topology.csi.vmware.com/openshift-region\",\"value\":\"vanderlab\"} {\"key\":\"topology.csi.vmware.com/openshift-zone\",\"value\":\"zone-b\"}" "rvanderp5-dev-rmlzp-worker-2-b5bnr {\"key\":\"topology.kubernetes.io/region\",\"value\":\"vanderlab\"} {\"key\":\"topology.kubernetes.io/zone\",\"value\":\"zone-c\"} {\"key\":\"topology.csi.vmware.com/openshift-region\",\"value\":\"vanderlab\"} {\"key\":\"topology.csi.vmware.com/openshift-zone\",\"value\":\"zone-c\"}"
Infra resource
apiVersion: config.openshift.io/v1 kind: Infrastructure metadata: creationTimestamp: "2023-01-23T15:43:19Z" generation: 1 name: cluster resourceVersion: "526" uid: 06aaabd8-0fb8-4cd4-a9bf-52c6c4d5ab8f spec: cloudConfig: key: config name: cloud-provider-config platformSpec: type: VSphere vsphere: failureDomains: - name: zone1 region: vanderlab server: vcenter.vanderlab.net topology: computeCluster: /vanderlab/host/cluster1 datacenter: vanderlab datastore: /vanderlab/datastore/nvme folder: /vanderlab/vm/rvanderp5-dev-5dhsb networks: - vander-pg resourcePool: /vanderlab/host/cluster1/Resources zone: zone-a - name: zone2 region: vanderlab server: vcenter.vanderlab.net topology: computeCluster: /vanderlab/host/cluster2 datacenter: vanderlab datastore: /vanderlab/datastore/nvmenfs folder: /vanderlab/vm/rvanderp5-dev-5dhsb networks: - vander-pg resourcePool: /vanderlab/host/cluster2/Resources zone: zone-b - name: zone3 region: vanderlab server: vcenter.vanderlab.net topology: computeCluster: /vanderlab/host/cluster3 datacenter: vanderlab datastore: /vanderlab/datastore/nvmenfs folder: /vanderlab/vm/rvanderp5-dev-5dhsb networks: - vander-pg resourcePool: /vanderlab/host/cluster3/Resources zone: zone-c nodeNetworking: external: {} internal: {} vcenters: - datacenters: - vanderlab port: 443 server: vcenter.vanderlab.net status: apiServerInternalURI: https://api-int.rvanderp5-dev.vanderlab.net:6443 apiServerURL: https://api.rvanderp5-dev.vanderlab.net:6443 controlPlaneTopology: HighlyAvailable etcdDiscoveryDomain: "" infrastructureName: rvanderp5-dev-rmlzp infrastructureTopology: HighlyAvailable platform: VSphere platformStatus: type: VSphere vsphere: apiServerInternalIP: 192.168.100.200 apiServerInternalIPs: - 192.168.100.200 ingressIP: 192.168.100.201 ingressIPs: - 192.168.100.201
cloud config
apiVersion: v1 data: config: | [Global] secret-name = "vsphere-creds" secret-namespace = "kube-system" insecure-flag = "1" [VirtualCenter "vcenter.vanderlab.net"] port = "443" datacenters = "vanderlab" [Workspace] server = "vcenter.vanderlab.net" datacenter = "vanderlab" default-datastore = "nvme" folder = "/vanderlab/vm/rvanderp5-dev-rmlzp" [Labels] region = "openshift-region" zone = "openshift-zone" kind: ConfigMap metadata: creationTimestamp: "2023-01-23T15:44:17Z" name: cloud-provider-config namespace: openshift-config resourceVersion: "2048" uid: 878755c8-9dc2-4046-9b75-ca4ffdd37d8c
feature gates
$ oc get featuregate -o yaml apiVersion: v1 items: - apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: annotations: include.release.openshift.io/ibm-cloud-managed: "true" include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" release.openshift.io/create-only: "true" creationTimestamp: "2023-01-23T15:44:05Z" generation: 1 name: cluster ownerReferences: - apiVersion: config.openshift.io/v1 kind: ClusterVersion name: version uid: 31a8f377-fbc0-4290-96fd-c837e158591b resourceVersion: "1649" uid: 62ccd645-ce34-44b6-b65b-e7fc46ff761d spec: {} kind: List metadata: resourceVersion: ""
https://github.com/openshift/cluster-cloud-controller-manager-operator/pull/218 has merged, retesting with nightly
Opened a PR in the API repo and performed an installation to confirm that with no feature gates defined in the cluster, the infrastructure resource contains what was provided by the installer.
cco#277 looking good, just bumped the api in the installer and waiting on e2e before going for final review