Environment Details:

• Platform: Azure Red Hat OpenShift (ARO) 4.17.43
• Operator: Cluster Observability Operator (COO) v1.3.0
• Logging / Loki Operator: v6.2.5
• Component: Console Plugin / UI (Observe -> Logs)

Problem Description:
When a non-admin user (e.g., a Developer with `edit` or `view` access to a specific Project) attempts to filter logs by Pod in the "Observe -> Logs" UI, specific pods are missing from the dropdown list. This occurs even when searching for the exact pod name prefix.

However, the logs do exist and can be viewed if the user manually constructs the LogQL query.

This issue does not occur for Cluster Administrators, who can see the pods in the dropdown after a brief delay or UI toggle.

Steps to Reproduce:
1. Log in as a Non-Admin user restricted to a specific custom namespace.
2. Navigate to Observe -> Logs.
3. Ensure the correct Namespace is selected.
4. In the "Pods" filter dropdown, type a partial pod name (e.g., edi-).
5. Result: The dropdown shows an incomplete list of pods. The specific target pod is missing.
6. Verification: Manually switch to "Show Query" and enter {kubernetes_pod_name="XYZ"}. Run the query. The logs appear, confirming backend connectivity and RBAC are correct.

Root Cause Analysis (Network Trace Evidence):
We have analyzed the browser network requests for both Admin and Non-Admin users and identified the defect in the UI Plugin's data fetching strategy.

The UI appears to use a "Dual-Source" strategy to populate the Pods dropdown:

1. Primary Source (Loki Label Values):

• Request: GET .../loki/api/v1/label/kubernetes_pod_name/values
• Result: This endpoint returns an incomplete/sampled list for both Admin and Non-Admin users. In our trace, it returned only 4 pods, missing the target pod entirely. This confirms that relying solely on Loki label values is insufficient for accurate UI filtering.

2. Secondary Source (Kubernetes API Fallback) - THE FAILURE POINT: The UI attempts to "backfill" the missing pods by querying the Kubernetes API directly.

Admin Behavior (Success):

• The Admin browser makes a request to: GET /api/kubernetes/api/v1/pods
• This returns a massive JSON payload (observed ~5MB) containing all pods in the cluster.
• The UI filters this list client-side and successfully populates the dropdown.

Non-Admin Behavior (Failure):

• The Non-Admin user never makes a request to the Kubernetes API.
• Because the user lacks cluster-admin permissions, the UI seemingly skips the global /api/v1/pods call.
• CRITICAL DEFECT: The UI fails to attempt a Namespaced-Scoped call (e.g., /api/kubernetes/api/v1/namespaces/ {namespace}/pods) as a fallback.
• Consequently, the non-admin user is left with only the incomplete list from Loki.

Impact:
Standard developers cannot use the UI to filter logs effectively. They are forced to manually type complex LogQL queries to find their pods, degrading the observability experience significantly.

Additional Notes:

• We observed generic TLS handshake error ... EOF logs in the plugin pod, but these appear to be unrelated network noise.
• There are no explicit JavaScript console errors, simply missing data in the fetch logic.

Request:
We request a fix for the Cluster Observability Operator UI Plugin. The logic must be updated to perform a Namespaced Pod List request when the user does not have permission to list all pods in the cluster.