Description of Problem

• Applications fail to sync with error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `kustomize build <path to cached source>/policygentemplates --enable-alpha-plugins` failed exit status 1: Error: /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator) /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator) : failure in plugin configured via /tmp/kust-plugin-config-482855201; exit status 1

Additional Info

• From openshift-gitops-repo-server- pod

  time="2024-10-10T06:27:01Z" level=info msg="manifest error cache hit: &ApplicationSource{RepoURL:https://gitlab.cee.example.com/demo/gitops-ztp.git,Path:policygentemplates,TargetRevision:rds-core-4.14,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/bb015cfdb75dbe2fc85290541d72b06a88feec53"
  time="2024-10-10T06:27:01Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `kustomize build <path to cached source>/policygentemplates --enable-alpha-plugins` failed exit status 1: Error: /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator)\n/.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator)\n: failure in plugin configured via /tmp/kust-plugin-config-482855201; exit status 1" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-10-10T06:27:01Z" grpc.time_ms=3.056 span.kind=server system=grpc
  time="2024-10-10T06:30:02Z" level=info msg="manifest error cache hit: &ApplicationSource{RepoURL:https://gitlab.cee.example.com/demo/gitops-ztp.git,Path:policygentemplates,TargetRevision:rds-core-4.14,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/bb015cfdb75dbe2fc85290541d72b06a88feec53"
  time="2024-10-10T06:30:02Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `kustomize build <path to cached source>/policygentemplates --enable-alpha-plugins` failed exit status 1: Error: /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator)\n/.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator)\n: failure in plugin configured via /tmp/kust-plugin-config-482855201; exit status 1" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-10-10T06:30:01Z" grpc.time_ms=1031.622 span.kind=server system=grpc
  time="2024-10-10T06:30:03Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:https://gitlab.cee.example.com/demo/gitops-ztp.git,Path:siteconfig,TargetRevision:rds-core-4.14,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/bb015cfdb75dbe2fc85290541d72b06a88feec53"
  time="2024-10-10T06:30:03Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-10-10T06:30:01Z" grpc.time_ms=2005.865 span.kind=server system=grpc
  

Problem Reproduction

• Deploy OpenShift-gitops operator `openshift-gitops-operator.v1.14.0`

Reproducibility

• So far 100%

Prerequisites/Environment

• Dualstack baremetal SNO hub cluster that deploys baremetal spoke cluster

Steps to Reproduce

• Follow GitOps-ZTP approach to deploy cluster

Expected Results

• Deployment succeeds

Actual Results

• Deployment fails as applications fail to sync