Epic Goal

• Red Hat Advanced Cluster Management for Kubernetes (RHACM) should, by default, trust the Openshift (OCP) internal CA for Services. While configuring the observability feature of RHACM to use an S3 bucket from ODF/Noobaa, I discovered that a custom CA must be defined.

Why is this important?

• Red Hat is encouraging customers to purchase OCP Platform Plus. This bundle includes, OCP, ODF, Quay, RHACM, and Advanced Cluster Security (ACS). This bundle will result in more customers leveraging ODF for their S3 storage needs. This makes it critically important that we reduce any friction associated with the use of our products. By accepting internal OCP certs by default we simplify the use of RHACM and ODF in a secure configuration.

Details

• $ cat secret-thanos-object-storage.yaml
  apiVersion: v1
  kind: Secret
  metadata:
    name: thanos-object-storage
    namespace: open-cluster-management-observability
  type: Opaque
  stringData:
    thanos.yaml: |
      type: s3
      config:
        bucket: acm-thanos
        endpoint: s3.openshift-storage.svc
        insecure: false
        access_key: NRjyj1I...
        secret_key: ajDxKey...
  
  
  $ oc get logs observability-thanos-store-shard-1-0
  ...
  err="Get \"https://s3.openshift-storage.svc/acm-thanos-711dcc8c-a878-4fe0-9fa4-d5d2bdfecf79/?location=\": x509: certificate signed by unknown authority\
  
  
  ## Certificate Authority data captured from a debug container/pod ##
  [root@debug /]# openssl s_client -connect s3.openshift-storage.svc:443 </dev/null 2>&1 </dev/null | head -20
  depth=1 CN = openshift-service-serving-signer@1648828414
  verify error:num=19:self signed certificate in certificate chain
  verify return:1
  depth=1 CN = openshift-service-serving-signer@1648828414
  verify return:1
  depth=0 CN = s3.openshift-storage.svc
  verify return:1
  CONNECTED(00000003)
  ---
  Certificate chain
   0 s:CN = s3.openshift-storage.svc
     i:CN = openshift-service-serving-signer@1648828414
   1 s:CN = openshift-service-serving-signer@1648828414
     i:CN = openshift-service-serving-signer@1648828414
  ...

• Similar details and conversation in #forum-acm-observability - https://coreos.slack.com/archives/CUU609ZQC/p1651698817536309

Acceptance Criteria

• ACM pods mount the internal service CA secrets and trust the CA - https://docs.openshift.com/container-platform/4.10/security/certificates/service-serving-certificate.html#add-service-certificate-configmap_service-serving-certificate

Open questions

1. Are there any drawbacks or negative issues if ACM trusts the OpenShift internal service CA