[Fluentd] Collector pods are failing to start with error system:serviceaccount:openshift-logging:logcollector cannot list resource pods in API group at the cluster scope

/usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:120:in `rescue in start_pod_watch': start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://kubernetes.default.svc: pods is forbidden: User "system:serviceaccount:openshift-logging:logcollector" cannot list resource "pods" in API group "" at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \\"system:serviceaccount:openshift-logging:logcollector\\" cannot list resource \\"pods\\" in API group \\"\\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403} (Fluent::ConfigError)
2022-11-10 05:04:06 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://kubernetes.default.svc: pods is forbidden: User \"system:serviceaccount:openshift-logging:logcollector\" cannot list resource \"pods\" in API group \"\" at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"pods is forbidden: User \\\"system:serviceaccount:openshift-logging:logcollector\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"pods\"},\"code\":403}\n)"

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance" 
  namespace: "openshift-logging"
spec:
  managementState: "Managed"  
  logStore:
    type: "elasticsearch"  
    retentionPolicy: 
      application:
        maxAge: 10h
      infra:
        maxAge: 10h
      audit:
        maxAge: 10h
    elasticsearch:
      nodeCount: 1 
      storage: {} 
      resources: 
          limits:
            memory: "4Gi"
          requests:
            memory: "1Gi"
      proxy: 
        resources:
          limits:
            memory: 256Mi
          requests:
            memory: 256Mi
      redundancyPolicy: "ZeroRedundancy"
  visualization:
    type: "kibana"  
    kibana:
      replicas: 1
  collection:
    logs:
      type: "fluentd"  
      fluentd: {}

$ oc get pods
NAME                                           READY   STATUS             RESTARTS      AGE
cluster-logging-operator-7f89b6cf9f-hj5vl      1/1     Running            0             17m
collector-gt6pc                                1/2     CrashLoopBackOff   7 (42s ago)   11m
collector-klzmc                                1/2     CrashLoopBackOff   7 (39s ago)   11m
collector-lx897                                1/2     CrashLoopBackOff   7 (48s ago)   11m
collector-ndf7q                                1/2     CrashLoopBackOff   7 (46s ago)   11m
collector-qrxrg                                1/2     CrashLoopBackOff   7 (47s ago)   11m
collector-v5nkm                                1/2     CrashLoopBackOff   7 (52s ago)   11m
elasticsearch-cdm-gh4en33p-1-bc896bbbb-xhslt   2/2     Running            0             11m
kibana-d9cd94fc5-nxvnr                         2/2     Running            0             11m
 
$ oc logs collector-v5nkm
Defaulted container "collector" out of: collector, logfilesmetricexporter
POD_IPS: 10.129.0.61, PROM_BIND_IP: 0.0.0.0
Setting each total_size_limit for 3 buffers to 54948160102 bytes
Setting queued_chunks_limit_size for each buffer to 6550
Setting chunk_limit_size for each buffer to 8388608
2022-11-10 05:04:05 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2022-11-10 05:04:05 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:8: warning: already initialized constant TRANSPORT_CLASS
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:3: warning: previous definition of TRANSPORT_CLASS was here
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:25: warning: already initialized constant SELECTOR_CLASS
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:20: warning: previous definition of SELECTOR_CLASS was here
#<Thread:0x00007fb65e148888 run> terminated with exception (report_on_exception is true):
/usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:120:in `rescue in start_pod_watch': start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://kubernetes.default.svc: pods is forbidden: User "system:serviceaccount:openshift-logging:logcollector" cannot list resource "pods" in API group "" at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \\"system:serviceaccount:openshift-logging:logcollector\\" cannot list resource \\"pods\\" in API group \\"\\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403} (Fluent::ConfigError)
)
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:111:in `start_pod_watch'
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:130:in `rescue in handle_exception': pods is forbidden: User "system:serviceaccount:openshift-logging:logcollector" cannot list resource "pods" in API group "" at the cluster scope (Kubeclient::HttpError)
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:120:in `handle_exception'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:135:in `get_pods_and_start_watcher'
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:112:in `start_pod_watch'
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response': 403 Forbidden (RestClient::Forbidden)
    from /usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
    from /usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
    from /usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
    from /usr/share/ruby/net/http.rb:933:in `start'
    from /usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
    from /usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
    from /usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
    from /usr/local/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:352:in `block in get_entities'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:121:in `handle_exception'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
    from /usr/local/share/gems/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:135:in `get_pods_and_start_watcher'
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:112:in `start_pod_watch'
    from /usr/local/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-3.1.2/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
2022-11-10 05:04:06 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://kubernetes.default.svc: pods is forbidden: User \"system:serviceaccount:openshift-logging:logcollector\" cannot list resource \"pods\" in API group \"\" at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"pods is forbidden: User \\\"system:serviceaccount:openshift-logging:logcollector\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"pods\"},\"code\":403}\n)