Without the ingress-controller, create the lb services and custom router deployment with custom secret. Can't get router stats from the prometheus sever

4.10.54

100%

1. 
% oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.54   True        False         7h26m   Cluster version is 4.10.54

2. Create a user student with password redhat123, then add cluster role to it
%oc adm policy add-cluster-role-to-user cluster-admin student

3. login with student
%oc get users
NAME          UID                                    FULL NAME   IDENTITIES
azure008      5476b64a-a3a1-47b8-8d46-4b0996210157               
student       eac42f5e-c6b8-4720-8359-b3e80966a89a               my_htpasswd_provider:student
testuser-46   0f9b4836-319f-42e4-a079-98f528d4379f               flexy-htpasswd-provider:testuser-46
% 
% oc login -u student -p redhat123
Login successful.


You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'


Using project "default".
%

4. In another terminal, follow the blow doc and add two secrets, router-template configmap, two lb services and router-shard-custom-oc-router deployment
https://docs.google.com/document/d/1N0mLNvIdHwVqTqnh9yjIX1Wj64LgEhW6jLKqe2mSD-Y/edit#

5. this step is for create the router-stats-shard-custom-oc-router secret
% echo student | base64
c3R1ZGVudAo=
% echo redhat123 | base64
cmVkaGF0MTIzCg==
% echo student | base64 > userbase64
% echo redhat123 | base64 > passwdbase64
% oc create secret generic router-stats-shard-custom-oc-router  --from-file=statsPassword=passwdbase64 --from-file=statsUsername=userbase64
secret/router-stats-shard-custom-oc-router created
%

6.
% oc -n openshift-ingress get pods -owide
NAME                                            READY   STATUS    RESTARTS   AGE     IP            NODE                                   NOMINATED NODE   READINESS GATES
router-default-7595664cfc-2hqqc                 1/1     Running   0          3h57m   10.131.0.14   shudi410a-r7d67-worker-eastus1-8t44d   <none>           <none>
router-default-7595664cfc-68xpv                 1/1     Running   0          3h57m   10.129.2.14   shudi410a-r7d67-worker-eastus2-9hmwg   <none>           <none>
router-shard-custom-oc-router-fd477dc9d-5nx29   1/1     Running   0          30m     10.129.2.18   shudi410a-r7d67-worker-eastus2-9hmwg   <none>           <none>
router-shard-custom-oc-router-fd477dc9d-xd7jf   1/1     Running   0          30m     10.131.0.18   shudi410a-r7d67-worker-eastus1-8t44d   <none>           <none>
%

7.
% oc -n openshift-ingress get secret | grep cus
router-certs-shard-custom-oc-router           kubernetes.io/tls                     2      37m
router-metrics-certs-shard-custom-oc-router   kubernetes.io/tls                     2      31m
router-stats-shard-custom-oc-router           Opaque                                2      33m
% 

8.
% oc -n openshift-ingress get secret router-stats-shard-custom-oc-router  -o go-template="{{.data.statsUsername}}" | base64 -d
c3R1ZGVudAo=
shudi@Shudis-MacBook-Pro Routercertificate % 
shudi@Shudis-MacBook-Pro Routercertificate % oc -n openshift-ingress get secret router-stats-shard-custom-oc-router  -o go-template="{{.data.statsPassword}}" | base64 -d
cmVkaGF0MTIzCg==
%

9. rsh to a pod, and try to get the stats with user c3R1ZGVudAo= and password cmVkaGF0MTIzCg==, but fails
% oc rsh centos-pod
sh-4.4# curl -u c3R1ZGVudAo=:cmVkaGF0MTIzCg== -kvvv https://10.129.2.18:1936/metrics
*   Trying 10.129.2.18...
* TCP_NODELAY set
* Connected to 10.129.2.18 (10.129.2.18) port 1936 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=router-internal-shard-custom-oc-router.openshift-ingress.svc
*  start date: Mar 23 09:08:12 2023 GMT
*  expire date: Mar 22 09:08:13 2025 GMT
*  issuer: CN=openshift-service-serving-signer@1679535203
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'c3R1ZGVudAo='
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /metrics HTTP/1.1
> Host: 10.129.2.18:1936
> Authorization: Basic YzNSMVpHVnVkQW89OmNtVmthR0YwTVRJekNnPT0=
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 23 Mar 2023 09:12:57 GMT
< Content-Length: 13
< 
Unauthorized
* Connection #0 to host 10.129.2.18 left intact
sh-4.4# exit
exit
%

10.
% oc -n openshift-ingress rsh router-shard-custom-oc-router-fd477dc9d-5nx29            
sh-4.4$ cd /var/lib/haproxy/conf/metrics-auth
sh-4.4$ ls
statsPassword  statsUsername
sh-4.4$ cat statsPassword | base64 -d
redhat123
sh-4.4$ cat statsUsername | base64 -d
student
sh-4.4$ cat statsPassword | base64 -d
redhat123
sh-4.4$ 

11. try to get router stats with user student and password redhat123, but fails
sh-4.4# curl -u student:redhat123 -kvvv https://10.129.2.18:1936/metrics
*   Trying 10.129.2.18...
* TCP_NODELAY set
* Connected to 10.129.2.18 (10.129.2.18) port 1936 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=router-internal-shard-custom-oc-router.openshift-ingress.svc
*  start date: Mar 23 09:08:12 2023 GMT
*  expire date: Mar 22 09:08:13 2025 GMT
*  issuer: CN=openshift-service-serving-signer@1679535203
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'student'
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /metrics HTTP/1.1
> Host: 10.129.2.18:1936
> Authorization: Basic c3R1ZGVudDpyZWRoYXQxMjM=
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 23 Mar 2023 09:11:05 GMT
< Content-Length: 13
< 
Unauthorized
* Connection #0 to host 10.129.2.18 left intact
sh-4.4#

12.
% oc get dnses.config cluster -oyaml | grep -i domain
  baseDomain: shudi410a.qe.azure.devcluster.openshift.com
%

13.
%  oc -n openshift-ingress get deployment router-shard-custom-oc-router -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2023-03-23T09:08:36Z"
  generation: 1
  labels:
    ingresscontroller.operator.openshift.io/owning-ingresscontroller: shard-custom-oc-router
  name: router-shard-custom-oc-router
  namespace: openshift-ingress
  resourceVersion: "178710"
  uid: 0b691a69-4cde-42bc-948c-b6522c5c85bc
spec:
  minReadySeconds: 30
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      ingresscontroller.operator.openshift.io/deployment-ingresscontroller: shard-custom-oc-router
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 50%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
        unsupported.do-not-use.openshift.io/override-liveness-grace-period-seconds: "10"
      creationTimestamp: null
      labels:
        ingresscontroller.operator.openshift.io/deployment-ingresscontroller: shard-custom-oc-router
        ingresscontroller.operator.openshift.io/hash: 7687fcc755
    spec:
      affinity:
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: ingresscontroller.operator.openshift.io/deployment-ingresscontroller
                  operator: In
                  values:
                  - shard-custom-oc-router
                - key: ingresscontroller.operator.openshift.io/hash
                  operator: NotIn
                  values:
                  - 7687fcc755
              topologyKey: kubernetes.io/hostname
            weight: 100
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: ingresscontroller.operator.openshift.io/deployment-ingresscontroller
                operator: In
                values:
                - shard-custom-oc-router
              - key: ingresscontroller.operator.openshift.io/hash
                operator: In
                values:
                - 7687fcc755
            topologyKey: kubernetes.io/hostname
      containers:
      - env:
        - name: DEFAULT_CERTIFICATE_DIR
          value: /etc/pki/tls/private
        - name: DEFAULT_DESTINATION_CA_PATH
          value: /var/run/configmaps/service-ca/service-ca.crt
        - name: RELOAD_INTERVAL
          value: 5s
        - name: ROUTER_ALLOW_WILDCARD_ROUTES
          value: "false"
        - name: ROUTER_CANONICAL_HOSTNAME
          value: router-shard-custom-oc-router.sec.shudi410a.qe.azure.devcluster.openshift.com
        - name: ROUTER_CIPHERS
          value: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        - name: ROUTER_CIPHERSUITES
          value: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        - name: ROUTER_DISABLE_HTTP2
          value: "true"
        - name: ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK
          value: "false"
        - name: ROUTER_LOAD_BALANCE_ALGORITHM
          value: leastconn
        - name: ROUTER_METRICS_TLS_CERT_FILE
          value: /etc/pki/tls/metrics-certs/tls.crt
        - name: ROUTER_METRICS_TLS_KEY_FILE
          value: /etc/pki/tls/metrics-certs/tls.key
        - name: ROUTER_METRICS_TYPE
          value: haproxy
        - name: ROUTER_SERVICE_NAME
          value: sharded
        - name: ROUTER_SERVICE_NAMESPACE
          value: openshift-ingress
        - name: ROUTER_SET_FORWARDED_HEADERS
          value: append
        - name: ROUTER_TCP_BALANCE_SCHEME
          value: source
        - name: ROUTER_THREADS
          value: "4"
        - name: ROUTE_LABELS
          value: type=sharded
        - name: SSL_MIN_VERSION
          value: TLSv1.2
        - name: STATS_PASSWORD_FILE
          value: /var/lib/haproxy/conf/metrics-auth/statsPassword
        - name: STATS_PORT
          value: "1936"
        - name: STATS_USERNAME_FILE
          value: /var/lib/haproxy/conf/metrics-auth/statsUsername
        - name: TEMPLATE_FILE
          value: /var/lib/haproxy/conf/custom/haproxy-config.template
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6434584560fb3f7b003b597adc1c0c546e2e1f76cb96cbd94a4665b50be434b7
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 1936
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: router
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 1936
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz/ready
            port: 1936
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
        startupProbe:
          failureThreshold: 120
          httpGet:
            path: /healthz/ready
            port: 1936
            scheme: HTTP
          periodSeconds: 1
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /etc/pki/tls/private
          name: default-certificate
          readOnly: true
        - mountPath: /var/run/configmaps/service-ca
          name: service-ca-bundle
          readOnly: true
        - mountPath: /var/lib/haproxy/conf/metrics-auth
          name: stats-auth
          readOnly: true
        - mountPath: /etc/pki/tls/metrics-certs
          name: metrics-certs
          readOnly: true
        - mountPath: /var/lib/haproxy/conf/custom
          name: template-volume
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: router
      serviceAccountName: router
      terminationGracePeriodSeconds: 3600
      topologySpreadConstraints:
      - labelSelector:
          matchExpressions:
          - key: ingresscontroller.operator.openshift.io/hash
            operator: In
            values:
            - 7687fcc755
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
      volumes:
      - name: default-certificate
        secret:
          defaultMode: 420
          secretName: router-certs-shard-custom-oc-router
      - configMap:
          defaultMode: 420
          items:
          - key: service-ca.crt
            path: service-ca.crt
          name: service-ca-bundle
          optional: false
        name: service-ca-bundle
      - name: stats-auth
        secret:
          defaultMode: 420
          secretName: router-stats-shard-custom-oc-router
      - name: metrics-certs
        secret:
          defaultMode: 420
          secretName: router-metrics-certs-shard-custom-oc-router
      - configMap:
          defaultMode: 420
          name: router-template
          optional: false
        name: template-volume
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2023-03-23T09:09:10Z"
    lastUpdateTime: "2023-03-23T09:09:10Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2023-03-23T09:08:36Z"
    lastUpdateTime: "2023-03-23T09:09:10Z"
    message: ReplicaSet "router-shard-custom-oc-router-fd477dc9d" has successfully
      progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2
%

can't get the router pod's stats 

can  get the router pod's stats