Uploaded image for project: 'Network Observability'
  1. Network Observability
  2. NETOBSERV-972

user authentication fails for non-kubeadmin users despite they're in cluster-admin groups

    • False
    • None
    • False
    • Hide
      Previously, when Network Observability was configured with "spec.loki.authToken" set to "DISABLED", a cluster administrator other than "kubeadmin" was not able to view network flows, getting authorization failure instead.

      Now, cluster adminitrators are able to view network flows as expected.
      Show
      Previously, when Network Observability was configured with "spec.loki.authToken" set to "DISABLED", a cluster administrator other than "kubeadmin" was not able to view network flows, getting authorization failure instead. Now, cluster adminitrators are able to view network flows as expected.
    • NetObserv - Sprint 234, NetObserv - Sprint 235, NetObserv - Sprint 236
    • Critical

      • Create a htpasswd IDP with users such as "testuser-0" etc.
      • Add "testuser-0" to cluster-admin or cluster-admins user group as:
        oc adm policy add-cluster-role-to-user cluster-admin testuser-0
        
      • Login using testuser-0
      • Deploy latest Downstream operator and 0-click loki , create flowcollector in netobserv NS with authToken: Disabled.
      • In Web console, navigate to Netflow traffic page, I am seeing queries are Unauthorized when logged in as testuser-0

      I verified users have cluster-admin role bindings, I tried assigning "cluster-admin" and "cluster-admins" role to the testuser-0 and also creating separate group and added testuser to that group and added cluster-admin role to the group but no luck, for e.g.:

      oc adm groups new mylocaladmins
      oc adm groups add-users mylocaladmins testuser-0
      oc adm policy add-cluster-role-to-group cluster-admin mylocaladmins
      


      in console plugin logs, I see:

      time="2023-03-31T18:03:55Z" level=debug msg="Checking auth: token found" module=handler.auth
      time="2023-03-31T18:03:55Z" level=debug msg="Checking authenticated user" module=handler.auth
      time="2023-03-31T18:03:55Z" level=debug msg="Checking auth: token found" module=handler.auth
      2023/03/31 18:03:55 http: TLS handshake error from 10.129.0.86:45846: EOF
      time="2023-03-31T18:03:55Z" level=debug msg="Checking authenticated user" module=handler.auth
      time="2023-03-31T18:03:55Z" level=debug msg="Checking auth: token found" module=handler.auth
      time="2023-03-31T18:04:09Z" level=debug msg="Checking authenticated user" module=handler.auth
      time="2023-03-31T18:04:09Z" level=debug msg="Checking auth: token found" module=handler.auth
      time="2023-03-31T18:04:09Z" level=debug msg="Checking authenticated user" module=handler.auth
      

            [NETOBSERV-972] user authentication fails for non-kubeadmin users despite they're in cluster-admin groups

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Important: Network observability 1.3.0 for Openshift), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2023:3905

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Important: Network observability 1.3.0 for Openshift), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:3905

            Mehul Modi added a comment -

            Verified, now with authToken: DISABLED a user would need to attach to at least view/edit/admin clusterRole or cluster-admin role to view flowlogs in UI. Also, verified by adding user to group with cluster-admin role.

            OCP: "4.13.0-0.nightly-2023-05-22-181752"
            NetObserv operator: v1.3.0
            Loki: 0-click-loki
            eBPF-agent: v1.3.0-36
            FLP: v1.3.0-36
            ConsolePlugin: v1.3.0-36

            Mehul Modi added a comment - Verified, now with authToken: DISABLED a user would need to attach to at least view/edit/admin clusterRole or cluster-admin role to view flowlogs in UI. Also, verified by adding user to group with cluster-admin role. OCP: "4.13.0-0.nightly-2023-05-22-181752" NetObserv operator: v1.3.0 Loki: 0-click-loki eBPF-agent: v1.3.0-36 FLP: v1.3.0-36 ConsolePlugin: v1.3.0-36

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of netobserv-midstream / Network Observability Operator Midstream on branch network-observability-1.3.0-rhel-9_ upstream _4ff286d64a77f3bfa8163557136d45f7 : Updated US source to: 2190369 NETOBSERV-972 check if cluster admin via namespaces (#326)

            Joel Takvorian added a comment - New PR: https://github.com/netobserv/network-observability-console-plugin/pull/326

            As discussed: we'll do the "list namespaces" hack again

            Joel Takvorian added a comment - As discussed: we'll do the "list namespaces" hack again

            Joel Takvorian added a comment - - edited

            Just to clarify my message above: the CVE we fixed in 1.1 is NOT reintroduced here, because we're still checking at least that the user is authenticated (even though they may be non-admins)

            Second clarification: the "hack" that users can do by entering the URL directly is not possible in our recommended configuration, ie. when loki-operator is used with FORWARD mode.

            Joel Takvorian added a comment - - edited Just to clarify my message above: the CVE we fixed in 1.1 is NOT reintroduced here, because we're still checking at least that the user is authenticated (even though they may be non-admins) Second clarification: the "hack" that users can do by entering the URL directly is not possible in our recommended configuration, ie. when loki-operator is used with FORWARD mode.

            Joel Takvorian added a comment - - edited

            rhn-support-memodi yes the intent of the PR was to relax checks so that in DISABLED authToken mode, the page is "normally" visible for cluster admins, with the downside that any authenticated users can hack it via entering the URL directly (this was the same situation in 1.0).
            As I commented here [1], we can reintroduce the hack that I did when fixing the CVE in 1.1, instead of relying on TokenReview. This would fix the remaining problem that you mention, although it's hacky (but we can be pragmatic sometimes and accept hacks) - basically the idea of the hack is to consider that any user able to list namespaces are cluster admins. We would then remove the TokenReview approach, which as we saw is more limited than what we expected at first.

            [1] https://github.com/netobserv/network-observability-console-plugin/pull/320#issuecomment-1496057096

            cc jpinsonn@redhat.com ocazade@redhat.com sounds good to you?

            Joel Takvorian added a comment - - edited rhn-support-memodi yes the intent of the PR was to relax checks so that in DISABLED authToken mode, the page is "normally" visible for cluster admins, with the downside that any authenticated users can hack it via entering the URL directly (this was the same situation in 1.0). As I commented here [1] , we can reintroduce the hack that I did when fixing the CVE in 1.1, instead of relying on TokenReview. This would fix the remaining problem that you mention, although it's hacky (but we can be pragmatic sometimes and accept hacks) - basically the idea of the hack is to consider that any user able to list namespaces are cluster admins. We would then remove the TokenReview approach, which as we saw is more limited than what we expected at first. [1] https://github.com/netobserv/network-observability-console-plugin/pull/320#issuecomment-1496057096 cc jpinsonn@redhat.com ocazade@redhat.com sounds good to you?

            Mehul Modi added a comment -

            jtakvori:
            As discussed in Scrum today, at the time of writing I had also tried to create user group name as "system:cluster-admins" (based of reading code here, added one of the users to the group and assigned cluster-admin role to the group (using instructions here) but that didn't work either.

            I am moving back to "In Progress" to decide on open question we have: whether we want to a check whether user is cluster-admin or belongs to group with cluster-admin RoleBinding when "authToken: Disabled"?

            Mehul Modi added a comment - jtakvori : As discussed in Scrum today, at the time of writing I had also tried to create user group name as "system:cluster-admins" (based of reading code here , added one of the users to the group and assigned cluster-admin role to the group ( using instructions here ) but that didn't work either. I am moving back to "In Progress" to decide on open question we have: whether we want to a check whether user is cluster-admin or belongs to group with cluster-admin RoleBinding when "authToken: Disabled"?

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of netobserv-midstream / Network Observability Operator Midstream on branch network-observability-1.3.0-rhel-9_ upstream _af69443d4dcfc95faec0532b1a749605 : Updated 4 upstream sources

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of netobserv-midstream / Network Observability Operator Midstream on branch network-observability-1.3.0-rhel-9_ upstream _af69443d4dcfc95faec0532b1a749605 : Updated 4 upstream sources

            Mehul Modi added a comment -

            jtakvori - I am verifying your PR # 320, so now it appears if authToken == Disabled, users don't have to be cluster-admins to view netflow-traffic page as long as they know about "/netflow-traffic" page, is this expected? I think in previous versions non-admin users used to get 404 when they directly access "/netflow-traffic" page (unless I am mis-remembering)

            in screenshot below you can see "Observe" menu is absent and I could access "/netflow-traffic" page directly, testuser-1 is not a cluster-admin.

            Mehul Modi added a comment - jtakvori - I am verifying your PR # 320, so now it appears if authToken == Disabled, users don't have to be cluster-admins to view netflow-traffic page as long as they know about "/netflow-traffic" page, is this expected? I think in previous versions non-admin users used to get 404 when they directly access "/netflow-traffic" page (unless I am mis-remembering) in screenshot below you can see "Observe" menu is absent and I could access "/netflow-traffic" page directly, testuser-1 is not a cluster-admin.

            Mehul Modi added a comment -

            jtakvori - While it may not be a release blocker, it's a Test Blocker. The UI tests in CI always run with non-kubeadmin user and in our tests we assign them in as cluster-admin role. I had tried adding that user in system:cluster-admins user group as well but that did not work either.

            when you say it works with FORWARD mode, did you have to set the according role binding to read logs or not?

            Correct, I had to create role bindings for FORWARD mode.

            Mehul Modi added a comment - jtakvori - While it may not be a release blocker, it's a Test Blocker. The UI tests in CI always run with non-kubeadmin user and in our tests we assign them in as cluster-admin role. I had tried adding that user in system:cluster-admins user group as well but that did not work either. when you say it works with FORWARD mode, did you have to set the according role binding to read logs or not? Correct, I had to create role bindings for FORWARD mode.

            After an investigation, yes I confirm this is an issue. The TokenReview request that we're doing doesn't seem sufficient to verify cluster-admins role, as it only list groups and not the associated roles (I had the wrong assumption that users with cluster-admin roles would automatically be added to the reserved system:cluster-admins group, but that's not the case)

            I think it's not a release 1.2 blocker, do you agree?

            rhn-support-memodi I have one question though: when you say it works with FORWARD mode, did you have to set the according role binding to read logs or not? (apiGroups: loki.grafana.com / resource "logs" ...) I'd be surprised if not.

            Joel Takvorian added a comment - After an investigation, yes I confirm this is an issue. The TokenReview request that we're doing doesn't seem sufficient to verify cluster-admins role, as it only list groups and not the associated roles (I had the wrong assumption that users with cluster-admin roles would automatically be added to the reserved system:cluster-admins group, but that's not the case) I think it's not a release 1.2 blocker, do you agree? rhn-support-memodi I have one question though: when you say it works with FORWARD mode, did you have to set the according role binding to read logs or not? (apiGroups: loki.grafana.com / resource "logs" ...) I'd be surprised if not.

            Mehul Modi added a comment -

            I tried with authToken: FORWARD mode and lokiStack, it works successfully with that, I wonder, if there's some bug or missing step with authToken==DISABLED?

            Mehul Modi added a comment - I tried with authToken: FORWARD mode and lokiStack, it works successfully with that, I wonder, if there's some bug or missing step with authToken==DISABLED?

              jtakvori Joel Takvorian
              rhn-support-memodi Mehul Modi
              Mehul Modi Mehul Modi
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: