https://issues.redhat.com/browse/NETOBSERV-1087 Support certificate verification in FLP ServiceMonitor when certificates are provided by the user Generate CA cert: * mkdir ~/certs * openssl genrsa -out rootCA.key 4096 * openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt <-- fill out the information, CN doesn't matter for CA cert. This should generate rootCA.crt and rootCA.key files. Generate server certificate and key: * openssl genrsa -out tls.key 2048 * Create CSR: openssl req -new -key tls.key -out tls.csr (Interactive mode) <-- fill out the information, CN here should be flowlogs-pipeline-prom.netobserv.svc * Prometheus needs SAN instead of CN, create below v3.ext file: $ cat v3.ext subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always basicConstraints = CA:TRUE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign subjectAltName = DNS:flowlogs-pipeline-prom.netobserv.svc issuerAltName = issuer:copy openssl x509 -req -in tls.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out tls.crt -days 500 -sha256 -extfile v3.ext * openssl x509 -req -in tls.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out tls.crt -days 500 -sha256 -extfile v3.ext These will generate tls.crt file: Reference: https://gist.github.com/fntlnz/cf14feb5a46b2eda428e000157447309 Reference For SAN: https://gist.github.com/KeithYeh/bb07cadd23645a6a62509b1ec8986bbc Create configmaps: oc create configmap prov-certs --from-file=tls.crt --from-file=tls.key oc create configmap prov-certs-ca --from-file rootCA.crt Update flowcollector to PROVIDED mode: tls: insecureSkipVerify: false provided: certFile: tls.crt certKey: tls.key name: prov-certs namespace: "" type: configmap providedCaFile: file: rootCA.crt name: prov-certs-ca namespace: "" type: configmap type: PROVIDED Check in new FLP pod logs, no errors are seen: $ oc logs pod/flowlogs-pipeline-j8b2q -f Starting flowlogs-pipeline: .... .... "MetricsSettings": "{\"nopanic\":true,\"port\":9102,\"prefix\":\"netobserv_\",\"tls\":{\"certpath\":\"/var/prom-certs/tls.crt\",\"keypath\":\"/var/prom-certs/tls.key\"}}", "Health": { "Address": "0.0.0.0", "Port": "8080" }, "Profile": { "Port": 6060 } } time=2023-09-11T16:03:14Z level=info msg=startServer: addr = :9102 time=2023-09-11T16:03:15Z level=info msg=connecting stages: grpc --> filter time=2023-09-11T16:03:15Z level=info msg=connecting stages: filter --> enrich time=2023-09-11T16:03:15Z level=info msg=connecting stages: enrich --> loki time=2023-09-11T16:03:15Z level=info msg=connecting stages: enrich --> prometheus time=2023-09-11T16:03:15Z level=info msg=starting PProf HTTP listener port=6060 ServiceMonitor: $ oc get servicemonitor/flowlogs-pipeline-monitor -o yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: creationTimestamp: "2023-09-11T13:32:57Z" generation: 11 labels: app: flowlogs-pipeline version: 92d689efbc61bc9f6a602196b82206f92cd92750a59612591fa5877da3d7ff4 name: flowlogs-pipeline-monitor namespace: netobserv ownerReferences: - apiVersion: flows.netobserv.io/v1beta1 blockOwnerDeletion: true controller: true kind: FlowCollector name: cluster uid: a7a1b9aa-7e66-4006-b0b1-21d398095642 resourceVersion: "95446" uid: e8689a20-8d65-4c44-b3ae-314db15ff535 spec: endpoints: - bearerTokenSecret: key: "" interval: 15s port: prometheus scheme: https tlsConfig: ca: configMap: key: rootCA.crt name: prov-certs-ca cert: {} serverName: flowlogs-pipeline-prom.netobserv.svc namespaceSelector: matchNames: - netobserv selector: matchLabels: app: flowlogs-pipeline On Targets console page, url: monitoring/targets?name=netobserv https://console-openshift-console.apps.memodi-09110850.qe.devcluster.openshift.com/monitoring/targets?name=netobserv All Targets should eventually transition as UP, note it may show DOWN as transient and NetObserv dashboards should have metrics populated. Verify prometheus has rootCA CM: $ oc get cm/prov-certs-ca -n openshift-monitoring NAME DATA AGE prov-certs-ca 1 67m Under openshift-monitoring NS: oc exec -it prometheus-k8s-0 -- sh sh-4.4$ ls -lrt /etc/prometheus/certs/*prov* lrwxrwxrwx. 1 root nobody 51 Sep 11 16:04 /etc/prometheus/certs/configmap_netobserv_prov-certs-ca_rootCA.crt -> ..data/configmap_netobserv_prov-certs-ca_rootCA.crt sh-4.4$ sh-4.4$ curl https://flowlogs-pipeline-prom.netobserv.svc:9102/metrics --cacert /etc/prometheus/certs/configmap_netobserv_prov-certs-ca_rootCA.crt -I HTTP/2 200 content-type: text/plain; version=0.0.4; charset=utf-8 date: Mon, 11 Sep 2023 16:53:14 GMT sh-4.4$ For deploymentModel == KAFKA: Updated SAN to include DNS:*.netobserv.svc $ cat v3.ext subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always basicConstraints = CA:TRUE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign subjectAltName = DNS:flowlogs-pipeline-prom.netobserv.svc,DNS:*.netobserv.svc issuerAltName = issuer:copy And regenerated tls.crt: openssl x509 -req -in tls.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out tls.crt -days 500 -sha256 -extfile v3.ext oc delete cm/prov-certs; oc create configmap prov-certs --from-file=tls.crt --from-file=tls.key Updated flowcollector: - deploymentModel == KAFKA and metrics.tls==DISABLED> - then metrics.tls==PROVIDED On Targets console page, url: monitoring/targets?name=netobserv https://console-openshift-console.apps.memodi-09110850.qe.devcluster.openshift.com/monitoring/targets?name=netobserv All Targets should eventually transition as UP, note it may show DOWN as transient and NetObserv dashboards should have metrics populated. $ oc exec -it prometheus-k8s-0 -- sh sh-4.4$ sh-4.4$ curl https://flowlogs-pipeline-prom.netobserv.svc:9102/metrics --cacert /etc/prometheus/certs/configmap_netobserv_prov-cert-ca-ns_rootCA.crt -I HTTP/2 200 content-type: text/plain; version=0.0.4; charset=utf-8 date: Wed, 13 Sep 2023 17:20:41 GMT sh-4.4$ Prom API for targets: api/prometheus/api/v1/targets?state=active --------- https://issues.redhat.com/browse/NETOBSERV-1295 $ oc get cm -n netobserv-cm NAME DATA AGE kube-root-ca.crt 1 141m openshift-service-ca.crt 1 141m prov-cert-ca-ns 1 96m prov-certs-ns 2 140m When certs are mounted in different NS: tls: insecureSkipVerify: false provided: certFile: tls.crt certKey: tls.key name: prov-certs-ns namespace: netobserv-cm type: configmap providedCaFile: file: rootCA.crt name: prov-cert-ca-ns namespace: netobserv-cm type: configmap type: PROVIDED (⎈ |memodi-091300940:openshift-monitoring)memodi@memodi-mac:/Users/memodi/certs $ oc get cm -n netobserv NAME DATA AGE console-plugin-config 1 10m flowlogs-pipeline-config 1 10m kube-root-ca.crt 1 147m loki-config 1 147m openshift-service-ca.crt 1 147m prov-cert-ca-ns 1 10m prov-certs-ns 2 10m $ oc get secrets/prometheus-k8s-tls-assets-0 -n openshift-monitoring -o yaml | egrep -i prov configmap_netobserv_prov-cert-ca-ns_rootCA.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdQekNDQkNlZ0F3SUJBZ0lVZllwQ0EvY2pESytMaXNpbjdXUzRJcG1YZmNRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2dhNHhDekFKQmdOVkJBWVRBbFZUTVJZd0ZBWURWUVFJREExTllYTnpZV05vZFhObGRIUnpNUTh3RFFZRApWUVFIREFaQ2IzTjBiMjR4RURBT0JnTlZCQW9NQjFKbFpDQklZWFF4RWpBUUJnTlZCQXNNQ1U5d1pXNVRhR2xtCmRERXVNQ3dHQTFVRUF3d2xabXh2ZDJ4dlozTXRjR2x3Wld4cGJtVXRjSEp2YlM1dVpYUnZZbk5sY25ZdWMzWmoKWXpFZ01CNEdDU3FHU0liM0RRRUpBUllSYldWdGIyUnBRSEpsWkdoaGRDNWpiMjB3SGhjTk1qTXdPVEV4TVRVMQpNVFUzV2hjTk1qWXdOekF4TVRVMU1UVTNXakNCcmpFTE1Ba0dBMVVFQmhNQ1ZWTXhGakFVQmdOVkJBZ01EVTFoCmMzTmhZMmgxYzJWMGRITXhEekFOQmdOVkJBY01Ca0p2YzNSdmJqRVFNQTRHQTFVRUNnd0hVbVZrSUVoaGRERVMKTUJBR0ExVUVDd3dKVDNCbGJsTm9hV1owTVM0d0xBWURWUVFERENWbWJHOTNiRzluY3kxd2FYQmxiR2x1WlMxdwpjbTl0TG01bGRHOWljMlZ5ZGk1emRtTmpNU0F3SGdZSktvWklodmNOQVFrQkZoRnRaVzF2WkdsQWNtVmthR0YwCkxtTnZiVENDQWlJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dJUEFEQ0NBZ29DZ2dJQkFMOFVjZGhkTVFJZk4xYmwKVlBxZzlXTXNIT1Fieno5OG45VTh0dU9pcWFybVZJanFRUW9yRU95NUVRb3k3YTFTeXgyVDExcCtObi9OMU5NaQpwaUQ4cSs0VWFoU2RTVTFEbmxKbVV0RzY1WFhCclhMblhLS0lqNjNrMFFweHpMMEMvUEtFL0lpd3VUelBzL0V4CmdoMU43Zm54SXlpT2E3c1ZsenVneHVYK2RMdWMybkt1V0JtRWZ0TlNmN2ZpdVhzY2VTQUxnMXZLUm1qTHFydkQKREVCR0tldmZDdW81cVZZMWQzcUp2cCs2QjJWSTFrNWRGa2pJcXBqaitCWVZsdy9Hd0tDTnhEK3hoUHBxZjlTcQprU2dNemhVTUcwOHlla3NXMm84YXRCQWJVYy9CL3lZV3VrSDB2S1lXdHVyMEVvcHpUNlV0ajlNS2djMjJEblI4CmhHSUtXa3l4V0hmQ0x6b3JHVCtPOHdBUXZVNmJOUkhKM1M1Uy9zdWZlVllFY2NOQTJ6VnlwR1Nlc3k3UTBuRUkKc05RLy90WjZaL2VQWVRnMzZCNDZXVDBtTXMxU1hhRFBOVUlIMkRZb2VaR2hWL2lMZkdweC9hY3J1ekVpeEhweQpGeWdrdGJaL0xlOXhjTWlmQnpLMzkzVXdoSEROb1lEejRSMlhIM0gvdVpTSFVxcXlrMHRjclRqSmZtWDJBaldPCitmSUFmY25NazBzRGxXWDBPU21PTi9WQVFDMmZ4VEluU25vZk1vN2NRWTQyNXlJRXVObE1aalhxWFBpSXI1UmIKS0N6cm14NUQ4Qkl0OWFhZVJnTEVZczRoR1FESlBDYkNGeTBzaWZBUkFGdDlHMHRnRE5ZUE9WdFZKSmd5U0lsbwpreE9TWWQzNVJqNkszM3phdjk0NGVyQWp5aVZMQWdNQkFBR2pVekJSTUIwR0ExVWREZ1FXQkJUbDBDQVJ4TlNUCnhlZEYxOEg0NVl0SEJ6MmRmakFmQmdOVkhTTUVHREFXZ0JUbDBDQVJ4TlNUeGVkRjE4SDQ1WXRIQnoyZGZqQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUNBUUJpUEl5REpmM0psZkFmSWNjdQpCRzJuNVBWU0w1SUdoY0tFeHh4NWs5QmQxU3ZpWDd2dndnd3VzdU5RbWhCYjZEMTN5Nkthc3oyd1BtR1FVUlVxCk5UN0NxS2EwRE1KR0NxZy9weE5SNzIzVGJ5bzl5ZnJMaUs4NllwWWhKaC9SSDJNcU9pcDR1RHdCS040ZVY3QUoKUmFTVHkzTlE4N3EyMDJNaUNURDRIaDFrYmVva0NGRlFpVklZR3ZQKzRNMkVScXQwbmFDQTZhVERnbitnV1lzSApLb2RjV0IyMndScER4aVVIZ3NjL3RnMUpRUUxCSGM4RTdrd0NiQ3NuQnhjbzdVMkJBSTlNVkZsNEtTOEN2T01lCldvQzk5T1hmY2xtczIyOTR2UWQ1dTdtUDduTUJwdU1UM1c5ZnM2YmFzdlltNGVNeE5qbThObzRTVjNjVlhaeU8KQjR0R0dHZ0pyeHl5Rk44WWxtdmYwMTNQdkRZVUFFd2hqM3FDemJCM2ZBMlFQaGVqTXpaOTJ6Y0x4RUpkb3Y0Rgo0UnBFT043TmUxSkdicUFUV2ZxTTFldjdJWVd0U25pVlMyTm1OMVBqTW1oUTdUNmUwL1ZueXVLQ1BSaElaNmQ5CjlMNlRUTTJ4MHNubXliR2JtOWQrLzhxRk5OQXhibjVRSHlOWlJPUkZqWDZWSU1mbkZ1aFIreEw2V3NBSmIvancKaHdRUDRGdm5xU3lZaktaZ2tqekdYU1pHNnhKK1MyRFlNdmpCL0dqdFBBVEV0L1k0UmlhRkVWa0JUeUx4NXVkZApzdStHbmxKUzlCUlAwdzhZaGhwMjRIVmc4dDlIZGZDTkxTdE5yNC94Z081a1JEcUhlemJmbEp5UU1USzZ4MStDClZoRlRkdUpLV29ndFJyUisydHNIZEVoSTF3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= https://console-openshift-console.apps.memodi-09110850.qe.devcluster.openshift.com/monitoring/targets?name=netobserv All Targets should eventually transition as UP, note it may show DOWN as transient and NetObserv dashboards should have metrics populated. sh-4.4$ curl https://flowlogs-pipeline-prom.netobserv.svc:9102/metrics --cacert /etc/prometheus/certs/configmap_netobserv_prov-cert-ca-ns_rootCA.crt -I HTTP/2 200 content-type: text/plain; version=0.0.4; charset=utf-8