Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-8204

ClusterCurator workflow template fails with hcp

XMLWordPrintable

    • False
    • None
    • False
    • No
    • Critical

      Description of problem:

      When creating an HCP cluster with a workflow template, the prehook pod fails with the following error:

      TASK [job_runner : Launch Instance Group] **************************************
      skipping: [localhost]
      
      TASK [job_runner : Read AnsibleJob Specs] **************************************
      skipping: [localhost]
      
      TASK [job_runner : Read AnsibleWorkflow Specs] *********************************
      An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ' raised while trying to get resource using (name=, namespace=, label_selectors=[], field_selectors=[])
      fatal: [localhost]: FAILED! => {"changed": false, "msg": "Exception '403\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'e7db5bc9-d56a-4b25-8bab-28e6dfd509fb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '79de5864-10ea-435c-a082-ea749a5366fc', 'X-Kubernetes-Pf-Prioritylevel-Uid': '07bfba6f-89e5-4d17-a815-5373b05327d2', 'Date': 'Mon, 16 Oct 2023 11:58:06 GMT', 'Content-Length': '406'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"ansibleworkflows.tower.ansible.com is forbidden: User \\\\\"system:serviceaccount:default:resource-operator-controller-manager-job\\\\\" cannot list resource \\\\\"ansibleworkflows\\\\\" in API group \\\\\"tower.ansible.com\\\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"group\":\"tower.ansible.com\",\"kind\":\"ansibleworkflows\"},\"code\":403}\\n'\nOriginal traceback: \n  File \"/usr/lib/python3.9/site-packages/kubernetes/dynamic/client.py\", line 55, in inner\n    resp = func(self, *args, **kwargs)\n\n  File \"/usr/lib/python3.9/site-packages/kubernetes/dynamic/client.py\", line 270, in request\n    api_response = self.client.call_api(\n\n  File \"/usr/lib/python3.9/site-packages/kubernetes/client/api_client.py\", line 348, in call_api\n    return self.__call_api(resource_path, method,\n\n  File \"/usr/lib/python3.9/site-packages/kubernetes/client/api_client.py\", line 180, in __call_api\n    response_data = self.request(\n\n  File \"/usr/lib/python3.9/site-packages/kubernetes/client/api_client.py\", line 373, in request\n    return self.rest_client.GET(url,\n\n  File \"/usr/lib/python3.9/site-packages/kubernetes/client/rest.py\", line 241, in GET\n    return self.request(\"GET\", url,\n\n  File \"/usr/lib/python3.9/site-packages/kubernetes/client/rest.py\", line 235, in request\n    raise ApiException(http_resp=r)\n' raised while trying to get resource using (name=, namespace=, label_selectors=[], field_selectors=[])"}
      
      PLAY RECAP *********************************************************************
      localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=5    rescued=0    ignored=0    

       

      Version-Release number of selected component (if applicable):

      ACM 2.9.0-DOWNSTREAM-2023-10-12-14-53-11

      AWS 4.13.0

      How reproducible:

      Always

      Steps to Reproduce:

      1. create cluster curator with workflow template as prehook
      2. create hcp with pausedUntil
      3. set clusterCuration to install

      Actual results:

      Expected results:

      Additional info:

      ClusterCurator:

      apiVersion: cluster.open-cluster-management.io/v1beta1
      kind: ClusterCurator
      metadata:
        creationTimestamp: "2023-10-16T11:54:05Z"
        generation: 7
        name: dhu-aap-aws-fips-02
        namespace: default
        resourceVersion: "3829031"
        uid: dfb00370-d210-45db-a37f-0a61c471f100
      spec:
        curatorJob: curator-job-hx9k4
        desiredCuration: install
        destroy:
          jobMonitorTimeout: 5
        install:
          jobMonitorTimeout: 5
          posthook:
          - extra_vars:
              cluster: hcp
              stage: posthook
              type: install
            name: Demo Workflow Template
            type: Workflow
          prehook:
          - extra_vars:
              cluster: hcp
              stage: prehook
              test: hypershift
            name: Demo Workflow Template
            type: Workflow
          towerAuthSecret: ans-tower
        scale:
          jobMonitorTimeout: 5
        upgrade:
          monitorTimeout: 120
          posthook:
          - extra_vars:
              cluster: hcp
              stage: posthook
              type: upgrade
            name: Auto_CLC_Sample_Template
            type: Job
          prehook:
          - extra_vars:
              cluster: hcp
              stage: prehook
              type: prehook
            name: Auto_CLC_Sample_Template
            type: Job
          towerAuthSecret: ans-tower
      status:
        conditions:
        - lastTransitionTime: "2023-10-16T11:57:49Z"
          message: 'curator-job-hx9k4 DesiredCuration: install'
          reason: Job_has_finished
          status: "False"
          type: clustercurator-job
        - lastTransitionTime: "2023-10-16T11:57:49Z"
          message: Executing init container prehook-ansiblejob
          reason: Job_has_finished
          status: "False"
          type: prehook-ansiblejob
        - lastTransitionTime: "2023-10-16T11:57:50Z"
          message: prehookjob-6c7cl
          reason: Job_has_finished
          status: "False"
          type: current-ansiblejob 

            fxiang@redhat.com Feng Xiang
            rhn-support-dhuynh David Huynh
            David Huynh David Huynh
            ACM QE Team
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: