Details
-
Bug
-
Resolution: Not a Bug
-
Minor
-
None
-
4.10
-
None
-
Moderate
-
No
-
Rejected
-
False
-
Description
Description of problem:
Without the ingress-controller, create the lb services and custom router deployment with custom secret. Can't get router stats from the prometheus sever
Version-Release number of selected component (if applicable):
4.10.54
How reproducible:
100%
Steps to Reproduce:
1.
% oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.10.54 True False 7h26m Cluster version is 4.10.54
2. Create a user student with password redhat123, then add cluster role to it
%oc adm policy add-cluster-role-to-user cluster-admin student
3. login with student
%oc get users
NAME UID FULL NAME IDENTITIES
azure008 5476b64a-a3a1-47b8-8d46-4b0996210157
student eac42f5e-c6b8-4720-8359-b3e80966a89a my_htpasswd_provider:student
testuser-46 0f9b4836-319f-42e4-a079-98f528d4379f flexy-htpasswd-provider:testuser-46
%
% oc login -u student -p redhat123
Login successful.
You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'
Using project "default".
%
4. In another terminal, follow the blow doc and add two secrets, router-template configmap, two lb services and router-shard-custom-oc-router deployment
https://docs.google.com/document/d/1N0mLNvIdHwVqTqnh9yjIX1Wj64LgEhW6jLKqe2mSD-Y/edit#
5. this step is for create the router-stats-shard-custom-oc-router secret
% echo student | base64
c3R1ZGVudAo=
% echo redhat123 | base64
cmVkaGF0MTIzCg==
% echo student | base64 > userbase64
% echo redhat123 | base64 > passwdbase64
% oc create secret generic router-stats-shard-custom-oc-router --from-file=statsPassword=passwdbase64 --from-file=statsUsername=userbase64
secret/router-stats-shard-custom-oc-router created
%
6.
% oc -n openshift-ingress get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-7595664cfc-2hqqc 1/1 Running 0 3h57m 10.131.0.14 shudi410a-r7d67-worker-eastus1-8t44d <none> <none>
router-default-7595664cfc-68xpv 1/1 Running 0 3h57m 10.129.2.14 shudi410a-r7d67-worker-eastus2-9hmwg <none> <none>
router-shard-custom-oc-router-fd477dc9d-5nx29 1/1 Running 0 30m 10.129.2.18 shudi410a-r7d67-worker-eastus2-9hmwg <none> <none>
router-shard-custom-oc-router-fd477dc9d-xd7jf 1/1 Running 0 30m 10.131.0.18 shudi410a-r7d67-worker-eastus1-8t44d <none> <none>
%
7.
% oc -n openshift-ingress get secret | grep cus
router-certs-shard-custom-oc-router kubernetes.io/tls 2 37m
router-metrics-certs-shard-custom-oc-router kubernetes.io/tls 2 31m
router-stats-shard-custom-oc-router Opaque 2 33m
%
8.
% oc -n openshift-ingress get secret router-stats-shard-custom-oc-router -o go-template="{{.data.statsUsername}}" | base64 -d
c3R1ZGVudAo=
shudi@Shudis-MacBook-Pro Routercertificate %
shudi@Shudis-MacBook-Pro Routercertificate % oc -n openshift-ingress get secret router-stats-shard-custom-oc-router -o go-template="{{.data.statsPassword}}" | base64 -d
cmVkaGF0MTIzCg==
%
9. rsh to a pod, and try to get the stats with user c3R1ZGVudAo= and password cmVkaGF0MTIzCg==, but fails
% oc rsh centos-pod
sh-4.4# curl -u c3R1ZGVudAo=:cmVkaGF0MTIzCg== -kvvv https://10.129.2.18:1936/metrics
* Trying 10.129.2.18...
* TCP_NODELAY set
* Connected to 10.129.2.18 (10.129.2.18) port 1936 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=router-internal-shard-custom-oc-router.openshift-ingress.svc
* start date: Mar 23 09:08:12 2023 GMT
* expire date: Mar 22 09:08:13 2025 GMT
* issuer: CN=openshift-service-serving-signer@1679535203
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'c3R1ZGVudAo='
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /metrics HTTP/1.1
> Host: 10.129.2.18:1936
> Authorization: Basic YzNSMVpHVnVkQW89OmNtVmthR0YwTVRJekNnPT0=
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 23 Mar 2023 09:12:57 GMT
< Content-Length: 13
<
Unauthorized
* Connection #0 to host 10.129.2.18 left intact
sh-4.4# exit
exit
%
10.
% oc -n openshift-ingress rsh router-shard-custom-oc-router-fd477dc9d-5nx29
sh-4.4$ cd /var/lib/haproxy/conf/metrics-auth
sh-4.4$ ls
statsPassword statsUsername
sh-4.4$ cat statsPassword | base64 -d
redhat123
sh-4.4$ cat statsUsername | base64 -d
student
sh-4.4$ cat statsPassword | base64 -d
redhat123
sh-4.4$
11. try to get router stats with user student and password redhat123, but fails
sh-4.4# curl -u student:redhat123 -kvvv https://10.129.2.18:1936/metrics
* Trying 10.129.2.18...
* TCP_NODELAY set
* Connected to 10.129.2.18 (10.129.2.18) port 1936 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=router-internal-shard-custom-oc-router.openshift-ingress.svc
* start date: Mar 23 09:08:12 2023 GMT
* expire date: Mar 22 09:08:13 2025 GMT
* issuer: CN=openshift-service-serving-signer@1679535203
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'student'
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /metrics HTTP/1.1
> Host: 10.129.2.18:1936
> Authorization: Basic c3R1ZGVudDpyZWRoYXQxMjM=
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 23 Mar 2023 09:11:05 GMT
< Content-Length: 13
<
Unauthorized
* Connection #0 to host 10.129.2.18 left intact
sh-4.4#
12.
% oc get dnses.config cluster -oyaml | grep -i domain
baseDomain: shudi410a.qe.azure.devcluster.openshift.com
%
13.
% oc -n openshift-ingress get deployment router-shard-custom-oc-router -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2023-03-23T09:08:36Z"
generation: 1
labels:
ingresscontroller.operator.openshift.io/owning-ingresscontroller: shard-custom-oc-router
name: router-shard-custom-oc-router
namespace: openshift-ingress
resourceVersion: "178710"
uid: 0b691a69-4cde-42bc-948c-b6522c5c85bc
spec:
minReadySeconds: 30
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
ingresscontroller.operator.openshift.io/deployment-ingresscontroller: shard-custom-oc-router
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 50%
type: RollingUpdate
template:
metadata:
annotations:
target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
unsupported.do-not-use.openshift.io/override-liveness-grace-period-seconds: "10"
creationTimestamp: null
labels:
ingresscontroller.operator.openshift.io/deployment-ingresscontroller: shard-custom-oc-router
ingresscontroller.operator.openshift.io/hash: 7687fcc755
spec:
affinity:
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: ingresscontroller.operator.openshift.io/deployment-ingresscontroller
operator: In
values:
- shard-custom-oc-router
- key: ingresscontroller.operator.openshift.io/hash
operator: NotIn
values:
- 7687fcc755
topologyKey: kubernetes.io/hostname
weight: 100
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: ingresscontroller.operator.openshift.io/deployment-ingresscontroller
operator: In
values:
- shard-custom-oc-router
- key: ingresscontroller.operator.openshift.io/hash
operator: In
values:
- 7687fcc755
topologyKey: kubernetes.io/hostname
containers:
- env:
- name: DEFAULT_CERTIFICATE_DIR
value: /etc/pki/tls/private
- name: DEFAULT_DESTINATION_CA_PATH
value: /var/run/configmaps/service-ca/service-ca.crt
- name: RELOAD_INTERVAL
value: 5s
- name: ROUTER_ALLOW_WILDCARD_ROUTES
value: "false"
- name: ROUTER_CANONICAL_HOSTNAME
value: router-shard-custom-oc-router.sec.shudi410a.qe.azure.devcluster.openshift.com
- name: ROUTER_CIPHERS
value: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
- name: ROUTER_CIPHERSUITES
value: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
- name: ROUTER_DISABLE_HTTP2
value: "true"
- name: ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK
value: "false"
- name: ROUTER_LOAD_BALANCE_ALGORITHM
value: leastconn
- name: ROUTER_METRICS_TLS_CERT_FILE
value: /etc/pki/tls/metrics-certs/tls.crt
- name: ROUTER_METRICS_TLS_KEY_FILE
value: /etc/pki/tls/metrics-certs/tls.key
- name: ROUTER_METRICS_TYPE
value: haproxy
- name: ROUTER_SERVICE_NAME
value: sharded
- name: ROUTER_SERVICE_NAMESPACE
value: openshift-ingress
- name: ROUTER_SET_FORWARDED_HEADERS
value: append
- name: ROUTER_TCP_BALANCE_SCHEME
value: source
- name: ROUTER_THREADS
value: "4"
- name: ROUTE_LABELS
value: type=sharded
- name: SSL_MIN_VERSION
value: TLSv1.2
- name: STATS_PASSWORD_FILE
value: /var/lib/haproxy/conf/metrics-auth/statsPassword
- name: STATS_PORT
value: "1936"
- name: STATS_USERNAME_FILE
value: /var/lib/haproxy/conf/metrics-auth/statsUsername
- name: TEMPLATE_FILE
value: /var/lib/haproxy/conf/custom/haproxy-config.template
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6434584560fb3f7b003b597adc1c0c546e2e1f76cb96cbd94a4665b50be434b7
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 1936
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: router
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 1936
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz/ready
port: 1936
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 256Mi
startupProbe:
failureThreshold: 120
httpGet:
path: /healthz/ready
port: 1936
scheme: HTTP
periodSeconds: 1
successThreshold: 1
timeoutSeconds: 1
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /etc/pki/tls/private
name: default-certificate
readOnly: true
- mountPath: /var/run/configmaps/service-ca
name: service-ca-bundle
readOnly: true
- mountPath: /var/lib/haproxy/conf/metrics-auth
name: stats-auth
readOnly: true
- mountPath: /etc/pki/tls/metrics-certs
name: metrics-certs
readOnly: true
- mountPath: /var/lib/haproxy/conf/custom
name: template-volume
readOnly: true
dnsPolicy: ClusterFirst
nodeSelector:
node-role.kubernetes.io/worker: ""
priorityClassName: system-cluster-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: router
serviceAccountName: router
terminationGracePeriodSeconds: 3600
topologySpreadConstraints:
- labelSelector:
matchExpressions:
- key: ingresscontroller.operator.openshift.io/hash
operator: In
values:
- 7687fcc755
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
volumes:
- name: default-certificate
secret:
defaultMode: 420
secretName: router-certs-shard-custom-oc-router
- configMap:
defaultMode: 420
items:
- key: service-ca.crt
path: service-ca.crt
name: service-ca-bundle
optional: false
name: service-ca-bundle
- name: stats-auth
secret:
defaultMode: 420
secretName: router-stats-shard-custom-oc-router
- name: metrics-certs
secret:
defaultMode: 420
secretName: router-metrics-certs-shard-custom-oc-router
- configMap:
defaultMode: 420
name: router-template
optional: false
name: template-volume
status:
availableReplicas: 2
conditions:
- lastTransitionTime: "2023-03-23T09:09:10Z"
lastUpdateTime: "2023-03-23T09:09:10Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2023-03-23T09:08:36Z"
lastUpdateTime: "2023-03-23T09:09:10Z"
message: ReplicaSet "router-shard-custom-oc-router-fd477dc9d" has successfully
progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 2
replicas: 2
updatedReplicas: 2
%
Actual results:
can't get the router pod's stats
Expected results:
can get the router pod's stats
Additional info: