Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-12376

Note that the hub kubeconfig needs to be valid before scaling clusters using the InfrastructureOperator

XMLWordPrintable

    • False
    • None
    • False
    • No

      Note: Doc team updates the current version of the documentation and the
      two previous versions (n-2), but we address *only high-priority, or
      customer-reported issues* for -2 releases in support.

      Describe the changes in the doc and link to your dev story:

      1. - [x] Mandatory: Add the required version to the Fix version/s field.

      2. - [x] Mandatory: Choose the type of documentation change or review.

      • [x] We need to update to an existing topic
      • [ ] We need to add a new document to an existing section
      • [ ] We need a whole new section; this is a function not
        documented before and doesn't belong in any current section
      • [ ] We need an Operator Advisory review and approval
      • [ ] We need a z-Stream (Errata) Advisory and Release note
        for MCE and/or ACM

      3. - [x] Mandatory: Use the following link to open the doc and find where the
      documentation update should go. Note: As the feature and doc is
      understood and developed, this placement decision may change:

      https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.10/html/clusters/cluster_mce_overview#add-nodes-ocp-infra-env

      4. - [x] Mandatory for GA content:

      • [x] Add steps, the diff, known issue, and/or other important
        conceptual information in the following space:

      If a user had changed the API certificate in a spoke cluster such that the kubeconfig created on install would now be invalid, that kubeconfig needs to be updated before nodes can be successfully added.

      # check if the kubeconfig related to the cluterdeployment is currently invalid
      
      $ export MY_KUBECONFIG_NAME=$(oc get cd $CLUSTERNAME -o "jsonpath={.spec.clusterMetadata.adminKubeconfigSecretRef.name}")
      $ oc extract secret/$MY_KUBECONFIG_NAME --keys=kubeconfig --to=- > my-kubeconfig-file
      $ oc --kubeconfig=my-kubeconfig-file get nodes
      Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority
      

      This output indicates an invalid kubeconfig secret that needs to be updated.

      Given that the new trusted CA bundle is in a file called "new-ca-certificate.pem" execute the following steps to update the kubeconfig secret:

      # Get base64 encoded cert bundle from existing kubeconfig certificate-authority-data field and decode it
      
      $ echo <base64 encoded blob> | base64 --decode > decoded-existing-certs.pem
      
      # create updated kubeconfig file as a copy of the original
      
      $ cp my-kubeconfig-file my-updated-kubeconfig-file
      
      # append new certs (in new-ca-certificate.pem) to the decoded pem
      
      $ cat decoded-existing-certs.pem new-ca-certificate.pem | openssl base64 -A
      
      # add base64 output from previous command as the value of the certificate-authority-data key in my-updated-kubeconfig-file using a text editor
      
      # check that the new kubeconfig is valid by querying the API using the new kubeconfig
      
      $ KUBECONFIG=my-updated-kubeconfig-file oc get nodes
      NAME            STATUS   ROLES                         AGE    VERSION
      extraworker-0   Ready    control-plane,master,worker   3d9h   v1.27.14+7852426
      extraworker-1   Ready    control-plane,master,worker   3d8h   v1.27.14+7852426
      extraworker-2   Ready    control-plane,master,worker   3d9h   v1.27.14+7852426
      extraworker-3   Ready    worker                        3d5h   v1.27.14+7852426
      
      # update the kubeconfig secret in the ACM hub cluster
      
      $ oc patch secret $MY_KUBECONFIG_NAME --type='json' -p="[{'op': 'replace', 'path': '/data/kubeconfig', 'value': '$(openssl base64 -A -in my-updated-kubeconfig-file)'},{'op': 'replace', 'path': '/data/raw-kubeconfig', 'value': '$(openssl base64 -A -in my-updated-kubeconfig-file)'}]"
      
      • [x] Add Required access level (example, *Cluster
        Administrator*) for the user to complete the task:

      Cluster Admin

      • [x] Add verification at the end of the task, how does the user
        verify success (a command to run or a result to see?)

      Verification was included in the above steps to determine if an update was needed and before the secret was edited.

      Additional Info:

      Most of this was taken in some way from https://github.com/openshift/hive/blob/73c38b4c889b3f12408ac572885346e04adf33a0/docs/troubleshooting.md#updating-certificate-authorities-within-the-admin-kubeconfig-secret so it's possible that we'll want to link to that doc in some way if that's our policy as indicated by Scott Berens' comment here https://redhat-internal.slack.com/archives/CTZTHFQRH/p1719325462357529?thread_ts=1719323775.150769&cid=CTZTHFQRH

              rh-ee-ofischer Oliver Fischer
              ncarboni@redhat.com Nick Carboni
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: