Manual installation of HyperShift cluster in the QE regular CI env according to the steps on https://hypershift.pages.dev/how-to/openstack -

• Install RHOS-17.1-RHEL-9-20241030.n.1
• Install 4.19.0-0.nightly-2025-03-09-063419:
  • Using IPv4 only
  • Using three workers
  • The masters flavor is `master`. The workers flavor is `m4.xlarge`:

    +--------------------------------------+-----------+-------+------+-----------+-------+-----------+
    | ID                                   | Name      |   RAM | Disk | Ephemeral | VCPUs | Is Public |
    +--------------------------------------+-----------+-------+------+-----------+-------+-----------+
    | 5666d803-f0da-4444-9ef2-a4d43b50f2ef | master    | 16384 |  100 |        10 |     4 | True      |
    | d6c76bb8-2c8b-473f-a932-8d7c9ee9e1bf | m4.xlarge | 16384 |   80 |         0 |     4 | True      |
    +--------------------------------------+-----------+-------+------+-----------+-------+-----------+ 
    

• Install the HyperShift and HCP CLI

  podman run --rm --privileged -it -v \
  $PWD:/output docker.io/library/golang:1.23 /bin/bash -c \
  'git clone https://github.com/openshift/hypershift.git && \
  cd hypershift/ && \
  make hypershift product-cli && \
  mv bin/hypershift /output/hypershift && \
  mv bin/hcp /output/hcp'
  
  sudo install -m 0755 -o root -g root $PWD/hypershift /usr/local/bin/hypershift
  sudo install -m 0755 -o root -g root $PWD/hcp /usr/local/bin/hcp
  rm $PWD/hypershift
  rm $PWD/hcp
  

• Deploy the HyperShift Operator:

  $ hypershift install --tech-preview-no-upgrade
  [stack@undercloud-0 ~]$ hypershift version
  Client Version: openshift/hypershift: 27db0fc33f91f7418d6bcc0ec98ecec70c2a23c0. Latest supported OCP: 4.19.0
  Server Version: <unknown>
  Server Supports OCP Versions: 4.19, 4.18, 4.17, 4.16, 4.15, 4.14
  

• Verify the HyperShift Operator

  [stack@undercloud-0 ~]$ oc get all -n hypershift
  Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
  NAME                            READY   STATUS    RESTARTS   AGE
  pod/operator-7744f7f4b4-kzs4x   1/1     Running   0          3m23s
  pod/operator-7744f7f4b4-ld8d2   1/1     Running   0          3m23s
  
  NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)            AGE
  service/operator   ClusterIP   172.30.5.101   <none>        9393/TCP,443/TCP   3m23s
  
  NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/operator   2/2     2            2           3m23s
  
  NAME                                  DESIRED   CURRENT   READY   AGE
  replicaset.apps/operator-7744f7f4b4   2         2         2       3m23s

• Create a dedicated 'shiftstack-hosted_cluster' project for the Hosted Cluster and update the clouds.yaml.
• Download and upload the latest RHCOS image in OpenStack (Not needed with the ORC changes)

  # Export OpenStack cloud
  export OS_CLOUD=shiftstack-hosted_cluster
  
  # Define the base URL
  BASE_URL="https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest"
  
  # Download the checksum file and OpenStack image
  sudo yum install -y wget
  wget -nv -O sha256sum.txt "${BASE_URL}/sha256sum.txt"
  wget -nv -O rhcos-openstack.x86_64.qcow2.gz "${BASE_URL}/rhcos-openstack.x86_64.qcow2.gz"
  
  # Verify the image integrity
  grep rhcos-openstack.x86_64.qcow2.gz sha256sum.txt | sha256sum -c -
  
  # Extract the image
  gunzip -f rhcos-openstack.x86_64.qcow2.gz
  
  # Upload to OpenStack
  openstack image create --disk-format qcow2 --file rhcos-openstack.x86_64.qcow2 rhcos
  

• Create a floating IP for the HostedCluster Ingress

  openstack floating ip create nova
  

• Setup a DNS server and update the domains for the ingress for both the management and the hosted clusters
  Note: For some reason, it didn't work from the undercloud. I'm using the hypervisor as the DNS server

sudo yum install -y bind-utils podman

#!/bin/bash
set -e  # Exit on error

### Configuration Variables ###
APPS_OSTEST_IP="10.46.44.164"  # IP for *.apps.ostest.shiftstack.local
APPS_HYPERSHIFT_IP="10.46.44.167"  # IP for *.apps.shiftstack-hcp.hypershift.lab
SERVER_IP=$(ip -4 -o addr show dev external | awk '{print $4}' | cut -d/ -f1)  # Detect the external IP

echo "🔍 Verifying existing services..."
sudo ss -tulnp | grep :53 || true
podman ps -a | grep dnsmasq || true
sudo systemctl status dnsmasq || true
sudo systemctl disable --now dnsmasq || true

echo "📂 Creating dnsmasq configuration..."
mkdir -p ~/dnsmasq/logs
cat <<EOF > ~/dnsmasq/dnsmasq.conf
port=53
listen-address=${SERVER_IP},127.0.0.1
bind-interfaces
address=/apps.ostest.shiftstack.local/${APPS_OSTEST_IP}
address=/apps.shiftstack-hcp.hypershift.lab/${APPS_HYPERSHIFT_IP}
resolv-file=/etc/resolv.conf
log-queries
log-facility=/var/log/dnsmasq/dnsmasq.log
EOF
chmod 644 ~/dnsmasq/dnsmasq.conf

echo "🚀 Starting dnsmasq in Podman..."
sudo podman run -d --name dnsmasq \
    --restart=always \
    --network=host \
    --cap-add=NET_ADMIN \
    -v ~/dnsmasq/dnsmasq.conf:/etc/dnsmasq.conf:Z \
    -v ~/dnsmasq/logs:/var/log/dnsmasq:Z \
    docker.io/alpine sh -c "apk add --no-cache dnsmasq && dnsmasq -k"

echo "✅ Verifying dnsmasq is running..."
sudo ss -tulnp | grep :53
podman ps -a
podman logs dnsmasq | tail -n 20

echo "🔥 Configuring firewall..."
sudo firewall-cmd --add-service=dns --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --list-services

echo "🔎 Testing DNS resolution..."
nslookup apps.ostest.shiftstack.local 127.0.0.1
dig @${SERVER_IP} apps.ostest.shiftstack.local

echo "✅ Setup complete! Configure other servers to use ${SERVER_IP} as their DNS server."

• Create the script to create the Hosted Cluster:

  [stack@undercloud-0 ~]$ cat create_hcp.sh
  #!/bin/bash
  #
  # create_hcp.sh
  
  export CLUSTER_NAME=shiftstack-hcp
  export BASE_DOMAIN=hypershift.lab
  export PUBLIC_NETWORK=nova
  export WORKER_COUNT="2"
  export IMAGE_NAME="rhcos"
  export FLAVOR="worker" # Flavor for the nodepool
  
  # OpenStack resources for the HostedCluster will be created in that project.
  export OS_CLOUD="shiftstack-hosted_cluster"
  
  oc get secret pull-secret -n openshift-config -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d > $HOME/pull-secret
  export PULL_SECRET="$HOME/pull-secret"
  
  # Optional flags:
  # External network to use for the Ingress endpoint.
  export EXTERNAL_NETWORK_ID="$(openstack network show $PUBLIC_NETWORK -c id -f value)"
  
  # SSH Key for the nodepool VMs
  export SSH_KEY="$HOME/.ssh/id_rsa.pub"
  
  # Create floating IP for Ingress and Update the DNS record
  export INGRESS_FLOATING_IP="10.46.44.162"
  
  export DNS_NAMESERVERS="10.46.44.129"
  
  hcp create cluster openstack \
    --name $CLUSTER_NAME \
    --base-domain $BASE_DOMAIN \
    --node-pool-replicas $WORKER_COUNT \
    --pull-secret $PULL_SECRET \
    --node-upgrade-type InPlace \
    --openstack-external-network-id $EXTERNAL_NETWORK_ID \
    --openstack-node-image-name $IMAGE_NAME \
    --openstack-node-flavor $FLAVOR \
    --control-plane-availability-policy SingleReplica \
    --infra-availability-policy SingleReplica \
    --ssh-key $SSH_KEY \
    --openstack-ingress-floating-ip $INGRESS_FLOATING_IP \
    --openstack-dns-nameservers $DNS_NAMESERVERS \
    --annotations=hypershift.openshift.io/skip-release-image-validation=true
  

• Create the HostedCluster on OpenStack

  [stack@undercloud-0 ~]$ sh create_hcp.sh
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"Namespace","namespace":"","name":"clusters"}
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"Secret","namespace":"clusters","name":"shiftstack-hcp-pull-secret"}
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"Secret","namespace":"clusters","name":"shiftstack-hcp-ssh-key"}
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"Secret","namespace":"clusters","name":"shiftstack-hcp-cloud-credentials"}
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"Role","namespace":"clusters","name":"capi-provider-role"}
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"Secret","namespace":"clusters","name":"shiftstack-hcp-etcd-encryption-key"}
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"","namespace":"clusters","name":"shiftstack-hcp"}
  {"level":"info","ts":"2025-03-11T07:39:26Z","msg":"Applied Kube resource","kind":"NodePool","namespace":"clusters","name":"shiftstack-hcp"}
  

• Verify the Hosted Control Plane (HCP)

  $ oc get --namespace clusters hostedclusters
  NAME             VERSION   KUBECONFIG                        PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
  shiftstack-hcp   4.18.4    shiftstack-hcp-admin-kubeconfig   Completed   True        False         The hosted control plane is available
  

  $ oc get nodepools.hypershift.openshift.io -n clusters
  NAME             CLUSTER          DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
  shiftstack-hcp   shiftstack-hcp   2               2               False         False        4.18.4    False             False
  

• Access the guest cluster and verify that it is healthy

  hcp create kubeconfig --name $CLUSTER_NAME > $CLUSTER_NAME-kubeconfig
  

  [stack@undercloud-0 ~]$ oc --kubeconfig $CLUSTER_NAME-kubeconfig get clusterversion
  NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
  version   4.18.4    True        False         144m    Cluster version is 4.18.4
  

  [stack@undercloud-0 ~]$ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes
  NAME                   STATUS   ROLES    AGE    VERSION
  shiftstack-hcp-l8jfj   Ready    worker   149m   v1.31.6
  shiftstack-hcp-sbshp   Ready    worker   149m   v1.31.6