Test on template versioned-installer-customer_vpc-disconnected_private_cluster-techpreview-ci, CAPI machine stuck in Pending phase, capa-controller-manager pod prints "failed to query AWSMachine instance by tags: failed to describe instances by tags: NoCredentialProviders: no valid providers in chain"

4.13.0-0.nightly-2022-12-23-223710

Always    

1.Create cluster, yaml file as below:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: huliu-awsdis27a-zzrc9
  namespace: openshift-cluster-api
spec:
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSCluster
    name: huliu-awsdis27a-zzrc9
    namespace: openshift-cluster-api
$ oc get cluster
NAME                    PHASE          AGE     VERSION
huliu-awsdis27a-zzrc9   Provisioning   3h29m     

2.apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster
metadata:
  name: huliu-awsdis27a-zzrc9
  namespace: openshift-cluster-api
spec:
  region: us-east-2
$ oc get awscluster      
NAME                    CLUSTER                 READY   VPC   BASTION IP
huliu-awsdis27a-zzrc9   huliu-awsdis27a-zzrc9   true 

3.apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
metadata:
  name: capi-aws-machine-template
  namespace: openshift-cluster-api
spec:
  template:
    spec:
      uncompressedUserData: true
      iamInstanceProfile: huliu-awsdis27a-zzrc9-worker-profile
      instanceType: m5.large
      failureDomain: us-east-2a
      cloudInit:
        insecureSkipSecretsManager: true
      ami:
        id: ami-09021f42f0e088c36
      subnet:
        id: subnet-051c9e4db9a27617e
      additionalSecurityGroups:
      - filters:
        - name: tag:Name
          values:
          - huliu-awsdis27a-zzrc9-worker-sg
$ oc get awsmachinetemplate
NAME                        AGE
capi-aws-machine-template   166m

4. Create machineset, yaml file as below:
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  name: capi-ms
  namespace: openshift-cluster-api
spec:
  clusterName: huliu-awsdis27a-zzrc9
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: huliu-awsdis27a-zzrc9
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: huliu-awsdis27a-zzrc9
    spec:
      bootstrap:
         dataSecretName: worker-user-data
      clusterName: huliu-awsdis27a-zzrc9
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: AWSMachineTemplate
        name: capi-aws-machine-template
$ oc get machine.cluster
NAME            CLUSTER                 NODENAME   PROVIDERID   PHASE          AGE    VERSION
capi-ms-9dsjt   huliu-awsdis27a-zzrc9                           Pending   176m

$ oc logs -f capa-controller-manager-d484bc464-kpn9r
 >
E1227 08:38:19.296140       1 controller.go:326] "Reconciler error" err=<
    failed to query AWSMachine instance by tags: failed to describe instances by tags: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" aWSMachine="openshift-cluster-api/capi-aws-machine-template-pg4dq" namespace="openshift-cluster-api" name="capi-aws-machine-template-pg4dq" reconcileID=36cf02e7-5932-4cb4-99a2-ff17961e61ae
E1227 08:39:43.734697       1 logger.go:84] "unable to find instance" err=<
    failed to query AWSMachine instance by tags: failed to describe instances by tags: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors     

Machine stuck in Pending phase   

Machine should go into Running phase