Description of problem:
governance-policy-framework fails to start post upgrade to 2.8 on some clusters
2023-05-08T14:27:18.349Z info setup logr@v1.2.3/logr.go:261 Using {"OperatorVersion": "0.0.1", "GoVersion": "go1.20.3", "GOOS": "linux", "GOARCH": "amd64"}
2023-05-08T14:27:18.351Z info setup logr@v1.2.3/logr.go:261 Starting lease controller to report status
2023-05-08T14:28:02.599Z info controller-runtime.metrics logr@v1.2.3/logr.go:261 Metrics server is starting to listen {"addr": "localhost:8383"}
2023-05-08T14:28:02.603Z error setup logr@v1.2.3/logr.go:279 unable to determine if Gatekeeper is installed {"error": "customresourcedefinitions.apiextensions.k8s.io \"constrainttemplates.templates.gatekeeper.sh\" is forbidden: User \"system:serviceaccount:open-cluster-management-agent-addon:governance-policy-framework-sa\" cannot list resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io \"open-cluster-management:governance-policy-framework-crd\" not found"}
github.com/go-logr/logr.Logger.Error
/remote-source/deps/gomod/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
main.addGkControllerToManager
/remote-source/app/main.go:617
main.getManager
/remote-source/app/main.go:429
main.main
/remote-source/app/main.go:268
runtime.main
/usr/lib/golang/src/runtime/proc.go:250
I0508
14:32:35.765953 1 round_trippers.go:443] POST
https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews
201 Created in 3 milliseconds2023/05/08 14:32:35 http: proxy error: dial tcp 127.0.0.1:8383: connect: connection refused2023/05/08 14:32:36 http: proxy error: dial tcp 127.0.0.1:8383: connect: connection refused2023/05/08 14:33:05 http: proxy error: dial tcp 127.0.0.1:8383: connect: connection refused
Version-Release number of selected component (if applicable):
2.8.0-DOWNSTREAM-2023-05-03-20-48-46 EC3-6
How reproducible: N/A
Steps to Reproduce:
Actual results:
addon is crashed and restarts continously
Expected results:
Pod starts successfully
Additional info:
I also noticed that policies that were created before upgrade got deleted. can they be related ?