Epic Goal

• Allow the user to customise the manifests generated by the installer.

In the IPI and UPI installation methods, the user can provide an install-config.yaml file and then run openshift-installer create manifests to generate the manifests that are added to k8s during bootstrapping. The generated manifests are spread across the manifests and openshift directories. The user then has the opportunity to edit the manifests before continuing to install with openshift-install create cluster (IPI) or openshift-install create ignition-configs (UPI).

In the agent installer, this create/edit loop is missing. Any manifests that appear in the openshift directory will be copied into the corresponding location by assisted-service before it generates the ignition-configs to install each node. This allows users to follow the usual instructions for adding a MachineConfig, which are to add the manifest to the openshift directory.

However, this mechanism cannot be used to override installer configs because there is a validation that checks that no two configs can attempt to create the same object in the k8s API. The only way to override them is to use the same filename, but there is no way to land files in the manifests directory, because only the openshift directory is scanned for manifests. (See OCPBUGS-3860 for an example of this.)

There is also no way to generate the manifests so that the user can see what configuration exists and make only the change they want. The installconfig.InstallConfig and agent.OptionalInstallConfig assets are separate assets that are created from the same file, but to generate the manifests a user would have to work around this by running openshift-installer create manifests; openshift-installer create install-config; openshift-installer agent create image. For now, this is probably a good thing since we don't want to generate all of the manifests locally and always use these to override the ones generated by the installer when it is run again by assisted-service. None of the assets generated in the openshift directory are ones we want to override; we only want to accept new manifests there. In the manifests directly, ideally we only want to transmit manifests that have been modified by the user to assisted-service, since otherwise the manifests generated at an earlier stage can override data edited at a later stage in the ZTP manifests - ideally in fact we would only transmit patches rather than the whole manifest.

This different InstallConfig assets also present a challenge to adding a new openshift-install agent create manifests command, since all of the assets that generate the existing manifests depend on installconfig.InstallConfig. The Agent InstallConfig also uses different (in most cases more permissive) validation than the regular InstallConfig. Some of this can be worked around by not generating the assets that rely on optional config (e.g. BareMetalHosts).

Manifests we might want to allow customisation of:

• manifests/cluster-ingress-02-config.yml
• manifests/cluster-network-02-config.yml
• manifests/cluster-proxy-01-config.yaml
• manifests/cluster-scheduler-02-config.yml
• manifests/cvo-overrides.yaml
• manifests/kube-cloud-config.yaml

Why is this important?

• Many configuration options for OpenShift are not captured in the install-config.yaml. In order to support the long tail of different day 1 configurations that are possible through IPI or UPI, the agent installer needs to surface a way for users to change the installation manifests.
• (Counterpoint: many of these changes are not actually needed on Day 1 and could be configured after the cluster is up, but users often want to install and configure their cluster in a single step.)

Scenarios

1. User wants to make the control plane nodes schedulable on an HA cluster.
2. User wants to edit the OVNKubernetes config (e.g. to enable jumbo frames).

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. What should the user interface to this look like, and what capabilities should it present?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>