Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29925

Hypershift hosted cluster deployment fails with OCP 4.15.0-rc.8 on IBM Power

    XMLWordPrintable

Details

    • No
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      I am trying to deploy Hypershift Agent-based Installation using ACM 2.10/MCE 2.5.0. When I use the `quay.io/openshift-release-dev/ocp-release:4.15.0-rc.8-multi` image, only 3 pods come up in the Hosted Cluster ns -

      # oc get po -n clusters-hypershift-017
NAME                                      READY   STATUS    RESTARTS   AGE
capi-provider-65cb859465-mbq7h            1/1     Running   0          47m
cluster-api-6c6bbd6dfb-px68k              1/1     Running   0          47m
control-plane-operator-85c4b57754-rxjsf   1/1     Running   0          47m

       

      And the control-plane-operator logs are as follows -

      # oc logs control-plane-operator-85c4b57754-rxjsf -n clusters-hypershift-017
{"level":"info","ts":"2024-02-26T12:21:25Z","msg":"Reconciling PKI","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-017","namespace":"clusters-hypershift-017"},"namespace":"clusters-hypershift-017","name":"hypershift-017","reconcileID":"13d2a6cc-2749-42a1-a980-d41bf727cf9c"}
{"level":"info","ts":"2024-02-26T12:21:25Z","msg":"Reconciling Control Plane PKI Operator","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-017","namespace":"clusters-hypershift-017"},"namespace":"clusters-hypershift-017","name":"hypershift-017","reconcileID":"13d2a6cc-2749-42a1-a980-d41bf727cf9c"}
{"level":"error","ts":"2024-02-26T12:21:26Z","msg":"Reconciler error","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-017","namespace":"clusters-hypershift-017"},"namespace":"clusters-hypershift-017","name":"hypershift-017","reconcileID":"13d2a6cc-2749-42a1-a980-d41bf727cf9c","error":"failed to update control plane: failed to reconcile control plane pki operator: failed to reconcile control plane pki operator role: roles.rbac.authorization.k8s.io \"control-plane-pki-operator\" is forbidden: user \"system:serviceaccount:clusters-hypershift-017:control-plane-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:clusters-hypershift-017\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"certificates.hypershift.openshift.io\"], Resources:[\"certificaterevocationrequests\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"certificates.hypershift.openshift.io\"], Resources:[\"certificaterevocationrequests/status\"], Verbs:[\"patch\"]}\n{APIGroups:[\"certificates.hypershift.openshift.io\"], Resources:[\"certificatesigningrequestapprovals\"], Verbs:[\"get\" \"list\" \"watch\"]}","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

      Following is the HC spec -

       

      spec:
  autoscaling: {}
  clusterID: 505d0908-598e-42de-9714-c9ac7aae40d2
  controllerAvailabilityPolicy: SingleReplica
  dns:
    baseDomain: ocp-powervs-ppc64le.com
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
        type: PersistentVolume
    managementType: Managed
  fips: false
  imageContentSources:
  - mirrors:
    - brew.registry.redhat.io
    source: registry.redhat.io
  - mirrors:
    - brew.registry.redhat.io
    source: registry.stage.redhat.io
  - mirrors:
    - brew.registry.redhat.io
    source: registry-proxy.engineering.redhat.com
  infraID: hypershift-017-klxzw
  infrastructureAvailabilityPolicy: SingleReplica
  issuerURL: https://kubernetes.default.svc
  networking:
    clusterNetwork:
    - cidr: 10.132.0.0/14
    networkType: OVNKubernetes
    serviceNetwork:
    - cidr: 172.31.0.0/16
  olmCatalogPlacement: management
  platform:
    agent:
      agentNamespace: clusters-hypershift-017
    type: Agent
  pullSecret:
    name: hypershift-017-pull-secret
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.15.0-rc.8-multi
  secretEncryption:
    aescbc:
      activeKey:
        name: hypershift-017-etcd-encryption-key
    type: aescbc
  services:
  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: OIDC
    servicePublishingStrategy:
      type: None
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  - service: OVNSbDb
    servicePublishingStrategy:
      type: Route
  sshKey:
    name: hypershift-017-ssh-key

       

       

      Version-Release number of selected component (if applicable):

      OCP 4.15.0-rc.8

      How reproducible:

      Always

      Steps to Reproduce:

      1. Installed ACM (advanced-cluster-management.v2.10.0-19) operator and MCE (multicluster-engine.v2.5.0-17).
      2. Created an AgentServiceConfig with the following details: 
        export DB_VOLUME_SIZE="10Gi"
export FS_VOLUME_SIZE="100Gi"
export OCP_VERSION="4.15.0"
export ARCH="ppc64le" 
export OCP_RELEASE_VERSION=415.92.202312132107-0
export ISO_URL="https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos/pre-release/4.15.0-rc.0/rhcos-4.15.0-rc.0-ppc64le-live.ppc64le.iso"
export ROOT_FS_URL="https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos/pre-release/4.15.0-rc.0/rhcos-4.15.0-rc.0-ppc64le-live-rootfs.ppc64le.img"

        3. Created an OCP 4.15.0-ec.8 Hosted cluster

        hypershift create cluster agent \
    --name=${HOSTED_CLUSTER_NAME} \
    --pull-secret=${PULL_SECRET_FILE} \
    --agent-namespace=${HOSTED_CONTROL_PLANE_NAMESPACE} \
    --base-domain=${BASEDOMAIN} \
    --api-server-address=api.${HOSTED_CLUSTER_NAME}.${BASEDOMAIN} \
    --ssh-key /root/id_rsa.pub \
    --release-image=${OCP_RELEASE_IMAGE} --render > render.yaml

      Expected results:

      Tried with the `quay.io/openshift-release-dev/ocp-release:4.15.0-rc.0-multi` image and the deployment was successful. Following are the logs of control plane operator -

      {"level":"info","ts":"2024-02-26T11:32:18Z","msg":"Successfully reconciled","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-017","namespace":"clusters-hypershift-017"},"namespace":"clusters-hypershift-017","name":"hypershift-017","reconcileID":"96bde4bc-f11a-4247-a169-3676fa4c26f8"}

      Attachments

        Activity

          People

            dravicha Dharaneeshwaran Ravichandran
            aishwarya_kamat1 Aishwarya Kamat
            Liangquan Li Liangquan Li
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: