https://issues.redhat.com/browse/NETOBSERV-1087 Support certificate verification in FLP ServiceMonitor when certificates are provided by the user Generate CA cert: * mkdir ~/certs * openssl genrsa -out rootCA.key 4096 * openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt <-- fill out the information, CN doesn't matter for CA cert. This should generate rootCA.crt and rootCA.key files. Generate server certificate and key: * openssl genrsa -out tls.key 2048 * Create CSR: openssl req -new -key tls.key -out tls.csr (Interactive mode) <-- fill out the information, CN here should be flowlogs-pipeline-prom.netobserv.svc * Prometheus needs SAN instead of CN, create below v3.ext file: $ cat v3.ext subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always basicConstraints = CA:TRUE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign subjectAltName = DNS:flowlogs-pipeline-prom.netobserv.svc issuerAltName = issuer:copy openssl x509 -req -in tls.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out tls.crt -days 500 -sha256 -extfile v3.ext * openssl x509 -req -in tls.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out tls.crt -days 500 -sha256 -extfile v3.ext These will generate tls.crt file: Reference: https://gist.github.com/fntlnz/cf14feb5a46b2eda428e000157447309 Reference For SAN: https://gist.github.com/KeithYeh/bb07cadd23645a6a62509b1ec8986bbc Create configmaps: oc create configmap prov-certs --from-file=tls.crt --from-file=tls.key oc create configmap prov-certs-ca --from-file rootCA.crt Update flowcollector to PROVIDED mode: tls: insecureSkipVerify: false provided: certFile: tls.crt certKey: tls.key name: prov-certs namespace: "" type: configmap providedCaFile: file: rootCA.crt name: prov-certs-ca namespace: "" type: configmap type: PROVIDED Check in new FLP pod logs, no errors are seen: $ oc logs pod/flowlogs-pipeline-j8b2q -f Starting flowlogs-pipeline: .... .... "MetricsSettings": "{\"nopanic\":true,\"port\":9102,\"prefix\":\"netobserv_\",\"tls\":{\"certpath\":\"/var/prom-certs/tls.crt\",\"keypath\":\"/var/prom-certs/tls.key\"}}", "Health": { "Address": "0.0.0.0", "Port": "8080" }, "Profile": { "Port": 6060 } } time=2023-09-11T16:03:14Z level=info msg=startServer: addr = :9102 time=2023-09-11T16:03:15Z level=info msg=connecting stages: grpc --> filter time=2023-09-11T16:03:15Z level=info msg=connecting stages: filter --> enrich time=2023-09-11T16:03:15Z level=info msg=connecting stages: enrich --> loki time=2023-09-11T16:03:15Z level=info msg=connecting stages: enrich --> prometheus time=2023-09-11T16:03:15Z level=info msg=starting PProf HTTP listener port=6060 ServiceMonitor: $ oc get servicemonitor/flowlogs-pipeline-monitor -o yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: creationTimestamp: "2023-09-11T13:32:57Z" generation: 11 labels: app: flowlogs-pipeline version: 92d689efbc61bc9f6a602196b82206f92cd92750a59612591fa5877da3d7ff4 name: flowlogs-pipeline-monitor namespace: netobserv ownerReferences: - apiVersion: flows.netobserv.io/v1beta1 blockOwnerDeletion: true controller: true kind: FlowCollector name: cluster uid: a7a1b9aa-7e66-4006-b0b1-21d398095642 resourceVersion: "95446" uid: e8689a20-8d65-4c44-b3ae-314db15ff535 spec: endpoints: - bearerTokenSecret: key: "" interval: 15s port: prometheus scheme: https tlsConfig: ca: configMap: key: rootCA.crt name: prov-certs-ca cert: {} serverName: flowlogs-pipeline-prom.netobserv.svc namespaceSelector: matchNames: - netobserv selector: matchLabels: app: flowlogs-pipeline On Targets console page, url: monitoring/targets?name=netobserv https://console-openshift-console.apps.memodi-09110850.qe.devcluster.openshift.com/monitoring/targets?name=netobserv All Targets should eventually transition as UP, note it may show DOWN as transient and NetObserv dashboards should have metrics populated. Verify prometheus has rootCA CM: $ oc get cm/prov-certs-ca -n openshift-monitoring NAME DATA AGE prov-certs-ca 1 67m Under openshift-monitoring NS: oc exec -it prometheus-k8s-0 -- sh sh-4.4$ ls -lrt /etc/prometheus/certs/*prov* lrwxrwxrwx. 1 root nobody 51 Sep 11 16:04 /etc/prometheus/certs/configmap_netobserv_prov-certs-ca_rootCA.crt -> ..data/configmap_netobserv_prov-certs-ca_rootCA.crt sh-4.4$ sh-4.4$ curl https://flowlogs-pipeline-prom.netobserv.svc:9102/metrics --cacert /etc/prometheus/certs/configmap_netobserv_prov-certs-ca_rootCA.crt -I HTTP/2 200 content-type: text/plain; version=0.0.4; charset=utf-8 date: Mon, 11 Sep 2023 16:53:14 GMT sh-4.4$ For deploymentModel == KAFKA: Updated SAN to include DNS:*.netobserv.svc $ cat v3.ext subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always basicConstraints = CA:TRUE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign subjectAltName = DNS:flowlogs-pipeline-prom.netobserv.svc,DNS:*.netobserv.svc issuerAltName = issuer:copy And regenerated tls.crt: openssl x509 -req -in tls.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out tls.crt -days 500 -sha256 -extfile v3.ext oc delete cm/prov-certs; oc create configmap prov-certs --from-file=tls.crt --from-file=tls.key Updated flowcollector: - deploymentModel == KAFKA and metrics.tls==DISABLED> - then metrics.tls==PROVIDED On Targets console page, url: monitoring/targets?name=netobserv https://console-openshift-console.apps.memodi-09110850.qe.devcluster.openshift.com/monitoring/targets?name=netobserv All Targets should eventually transition as UP, note it may show DOWN as transient and NetObserv dashboards should have metrics populated. $ oc exec -it prometheus-k8s-0 -- sh sh-4.4$ sh-4.4$ curl https://flowlogs-pipeline-transformer-prom.netobserv.svc:9102/metrics --cacert /etc/prometheus/certs/configmap_netobserv_prov-certs-ca_rootCA.crt -I HTTP/2 200 content-type: text/plain; version=0.0.4; charset=utf-8 date: Mon, 11 Sep 2023 20:17:31 GMT sh-4.4$