https://issues.redhat.com/browse/NETOBSERV-773 Loki: - statusTls - gateway - user certs Kafka: - Kafka-cluster - server-ca.crt - user crt exporters: - Kafka-exporter - server-ca.crt - user crt FLP: - metrics server === flowcollector excerpt with kafka and Loki TLS configs: exporters: - kafka: address: kafka-cluster-kafka-bootstrap.netobserv-kafka tls: caCert: certFile: ca.crt name: kafka-cluster-cluster-ca-cert namespace: netobserv-kafka type: secret enable: true insecureSkipVerify: false userCert: certFile: user.crt certKey: user.key name: flp-kafka-export namespace: netobserv-kafka type: secret topic: network-flows-export type: KAFKA kafka: address: kafka-cluster-kafka-bootstrap.netobserv-kafka tls: caCert: certFile: ca.crt name: kafka-cluster-cluster-ca-cert namespace: netobserv-kafka type: secret enable: true insecureSkipVerify: false userCert: certFile: user.crt certKey: user.key name: flp-kafka namespace: netobserv-kafka type: secret topic: network-flows loki: authToken: FORWARD batchSize: 10485760 batchWait: 1s maxBackoff: 5s maxRetries: 2 minBackoff: 1s staticLabels: app: netobserv-flowcollector statusTls: caCert: certFile: service-ca.crt name: loki-ca-bundle namespace: netobserv-loki type: configmap enable: true insecureSkipVerify: false userCert: certFile: tls.crt certKey: tls.key name: loki-query-frontend-http namespace: netobserv-loki type: secret statusUrl: https://loki-query-frontend-http.netobserv-loki.svc:3100/ tenantID: netobserv timeout: 10s tls: caCert: certFile: service-ca.crt name: loki-gateway-ca-bundle namespace: netobserv-loki type: configmap enable: true insecureSkipVerify: false userCert: namespace: "" url: https://loki-gateway-http.netobserv-loki.svc.cluster.local:8080/api/logs/v1/network Copied CM in netobserv NS: $ oc get cm NAME DATA AGE console-plugin-config 1 4h35m flowlogs-pipeline-transformer-config 1 77m kube-root-ca.crt 1 4h35m loki-ca-bundle 1 145m loki-config 1 4h35m loki-gateway-ca-bundle 1 3h45m openshift-service-ca.crt 1 4h35m Copied secrets in netobserv NS: $ oc get secrets -n netobserv NAME TYPE DATA AGE builder-dockercfg-qzbqf kubernetes.io/dockercfg 1 5h31m builder-token-sp792 kubernetes.io/service-account-token 4 5h31m console-serving-cert kubernetes.io/tls 2 5h30m default-dockercfg-v8k4s kubernetes.io/dockercfg 1 5h31m default-token-cgmkh kubernetes.io/service-account-token 4 5h31m deployer-dockercfg-ggw8g kubernetes.io/dockercfg 1 5h31m deployer-token-lq5bm kubernetes.io/service-account-token 4 5h31m flowlogs-pipeline-transformer-dockercfg-bqdn9 kubernetes.io/dockercfg 1 133m flowlogs-pipeline-transformer-token-jt8mn kubernetes.io/service-account-token 4 133m flp-kafka Opaque 5 133m flp-kafka-export Opaque 5 60m kafka-cluster-cluster-ca-cert Opaque 3 133m loki-query-frontend-http kubernetes.io/tls 2 3h25m netobserv-plugin-dockercfg-zvvms kubernetes.io/dockercfg 1 5h30m netobserv-plugin-token-x7k57 kubernetes.io/service-account-token 4 5h30m Copied CM in netobserv-privileged NS: $ oc get secrets -n netobserv-privileged NAME TYPE DATA AGE builder-dockercfg-5hf89 kubernetes.io/dockercfg 1 5h31m builder-token-whvld kubernetes.io/service-account-token 4 5h31m default-dockercfg-cw6xp kubernetes.io/dockercfg 1 5h31m default-token-gh9fg kubernetes.io/service-account-token 4 5h31m deployer-dockercfg-h64vr kubernetes.io/dockercfg 1 5h31m deployer-token-nnzdc kubernetes.io/service-account-token 4 5h31m flp-kafka Opaque 5 134m kafka-cluster-cluster-ca-cert Opaque 3 134m netobserv-ebpf-agent-dockercfg-c8ptw kubernetes.io/dockercfg 1 5h31m netobserv-ebpf-agent-token-zsmrw kubernetes.io/service-account-token 4 5h31m Verified kafka exports and ebpf-agent are able to ingest flows to kafka Topic and plugin is able to fetch from Loki. Kafka-exports were verified by logging on the container and looking at partitions because I could not get it kfk CLI to work with TLS certs. When tls == AUTO for FLP metrics, I get below error: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 45s default-scheduler Successfully assigned netobserv/flowlogs-pipeline-rrzrj to ip-10-0-132-145.us-east-2.compute.internal Warning FailedMount 14s (x7 over 46s) kubelet MountVolume.SetUp failed for volume "prom-certs" : secret "flowlogs-pipeline-prom" not found When tls == PROVIDED for FLP metrics: tls: provided: certFile: service-ca.crt name: openshift-service-ca.crt namespace: "" type: configmap type: PROVIDED FLP logs has error: time=2023-05-30T14:55:45Z level=info msg=startServer: addr = :9102 time=2023-05-30T14:55:45Z level=error msg=error in http.ListenAndServe: read /var/prom-certs/: is a directory time=2023-05-30T14:55:46Z level=info msg=connecting stages: grpc --> enrich time=2023-05-30T14:55:46Z level=info msg=connecting stages: enrich --> loki time=2023-05-30T14:55:46Z level=info msg=connecting stages: enrich --> prometheus time=2023-05-30T14:55:46Z level=info msg=starting PProf HTTP listener port=6060