-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
rhosdt-3.9
-
False
-
-
False
-
-
The current implementation of the OpenTelemetry Operator and CRD does not support collectorSelectors for the TargetAllocator CR.
This is mandatory as without any workload in the Cluster with a certain label will be assigned as collector depending on the RBAC permission of the serviceAccount.
The Env OTELCOL_NAMESPACE can limit the selected collectors but still in the example I end up with none OTC pods being assigned and therefor missing Scrape Targets.
Reproducer:
- create a namespace
oc create reproducer
- create a TargetAllocator
cat <<'EOF' | oc -n reproducer create -f- apiVersion: opentelemetry.io/v1alpha1 kind: TargetAllocator metadata: name: reproducer spec: allocationStrategy: consistent-hashing collectorNotReadyGracePeriod: 30s filterStrategy: relabel-config ipFamilyPolicy: SingleStack managementState: managed observability: metrics: enableMetrics: false prometheusCR: enabled: false scrapeInterval: 30s replicas: 1 scrapeConfigs: - fallback_scrape_protocol: PrometheusText1.0.0 job_name: reproducer scheme: http static_configs: - targets: - host01:9090 - host02:9090 - host03:9090 - host04:9090 - host05:9090 - host06:9090 - host07:9090 - host08:9090 - host09:9090 - host10:9090 - host11:9090 - host12:9090 - host13:9090 - host14:9090 - host15:9090 - host16:9090 - host17:9090 - host18:9090 - host19:9090 - host20:9090 EOF
- use port-forwarding to see which collectors have been assigned
oc -n reproducer port-forward service/reproducer-targetallocator 8080:80 & cat <<'EOF' | python import re, requests pattern = r'.*href="/debug/collector.+?>([^<]+)</a>' rsp = requests.get("http://localhost:8080").text for match in re.finditer(pattern, rsp): print(match.groups()[0]) EOF
- will output the targetAllocator itself and or multiple other pods in that namespace
- add a simple httpd pod to verify targetAllocator selects it for target dispatching
cat <<'EOF' | oc -n reproducer create -f- apiVersion: v1 kind: Pod metadata: name: example labels: app: httpd namespace: targetallocatorreproducer spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: httpd image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest' ports: - containerPort: 8080 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL EOF
- re-check the targetAllocator collector list
oc -n reproducer port-forward service/reproducer-targetallocator 8080:80 & cat <<'EOF' | python import re, requests pattern = r'.*href="/debug/collector.+?>([^<]+)</a>' rsp = requests.get("http://localhost:8080").text for match in re.finditer(pattern, rsp): print(match.groups()[0]) EOF
- this for example outputs now (after adding two httpd pods)
example example2 reproducer-targetallocator-75596bd594-mp4np
- the UI of the targetAllocator also shows assigned Job and TargetCounts
Collector Job Count Target Count example 1 6 example2 1 10 reproducer-targetallocator-75596bd594-mp4np 1 4
- impacts account
-
TRACING-5975 RHOSDT Hackaton 3.9
-
- To Do
-