Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-10887

oauth-server fails to invalidate cache, causing non existing groups being referenced

XMLWordPrintable

    • Critical
    • Auth - Sprint 235
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Customer Escalated

      Description of problem:

      Following https://bugzilla.redhat.com/show_bug.cgi?id=2102765 respectively https://issues.redhat.com/browse/OCPBUGS-2140 problems with OpenID Group sync have been resolved.
      
      Yet the problem documented in https://bugzilla.redhat.com/show_bug.cgi?id=2102765 still does exist and we see that Groups that are being removed are still part of the chache in oauth-apiserver, causing a panic of the respective components and failures during login for potentially affected users.
      
      So in general, it looks like that oauth-apiserver cache is not properly refreshing or handling the OpenID Groups being synced.
      
      E1201 11:03:14.625799       1 runtime.go:76] Observed a panic: interface conversion: interface {} is nil, not *v1.Group
      goroutine 3706798 [running]:
      k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()
          k8s.io/apiserver@v0.22.2/pkg/server/filters/timeout.go:103 +0xb0
      panic({0x1aeab00, 0xc001400390})
          runtime/panic.go:838 +0x207
      k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1.1.1()
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/audit.go:80 +0x2a
      k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1.1()
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/audit.go:89 +0x250
      panic({0x1aeab00, 0xc001400390})
          runtime/panic.go:838 +0x207
      github.com/openshift/library-go/pkg/oauth/usercache.(*GroupCache).GroupsFor(0xc00081bf18?, {0xc000c8ac03?, 0xc001400360?})
          github.com/openshift/library-go@v0.0.0-20211013122800-874db8a3dac9/pkg/oauth/usercache/groups.go:47 +0xe7
      github.com/openshift/oauth-server/pkg/groupmapper.(*UserGroupsMapper).processGroups(0xc0002c8880, {0xc0005d4e60, 0xd}, {0xc000c8ac03, 0x7}, 0x1?)
          github.com/openshift/oauth-server/pkg/groupmapper/groupmapper.go:101 +0xb5
      github.com/openshift/oauth-server/pkg/groupmapper.(*UserGroupsMapper).UserFor(0xc0002c8880, {0x20f3c40, 0xc000e18bc0})
          github.com/openshift/oauth-server/pkg/groupmapper/groupmapper.go:83 +0xf4
      github.com/openshift/oauth-server/pkg/oauth/external.(*Handler).login(0xc00022bc20, {0x20eebb0, 0xc00041b058}, 0xc0015d8200, 0xc001438140?, {0xc0000e7ce0, 0x150})
          github.com/openshift/oauth-server/pkg/oauth/external/handler.go:209 +0x74f
      github.com/openshift/oauth-server/pkg/oauth/external.(*Handler).ServeHTTP(0xc00022bc20, {0x20eebb0, 0xc00041b058}, 0x0?)
          github.com/openshift/oauth-server/pkg/oauth/external/handler.go:180 +0x74a
      net/http.(*ServeMux).ServeHTTP(0x1c9dda0?, {0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          net/http/server.go:2462 +0x149
      github.com/openshift/oauth-server/pkg/server/headers.WithRestoreAuthorizationHeader.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          github.com/openshift/oauth-server/pkg/server/headers/oauthbasic.go:27 +0x10f
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x20eebb0?, 0xc00041b058?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:103 +0x1a5
      net/http.HandlerFunc.ServeHTTP(0xc0005e0280?, {0x20eebb0?, 0xc00041b058?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/authorization.go:64 +0x498
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x20eebb0?, 0xc00041b058?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:79 +0x178
      net/http.HandlerFunc.ServeHTTP(0x2f6cea0?, {0x20eebb0?, 0xc00041b058?}, 0x3?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/server/filters/maxinflight.go:187 +0x2a4
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x20eebb0?, 0xc00041b058?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:103 +0x1a5
      net/http.HandlerFunc.ServeHTTP(0x11?, {0x20eebb0?, 0xc00041b058?}, 0x1aae340?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/impersonation.go:50 +0x21c
      net/http.HandlerFunc.ServeHTTP(0xc000d52120?, {0x20eebb0?, 0xc00041b058?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:79 +0x178
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x20eebb0?, 0xc00041b058?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x20eebb0, 0xc00041b058}, 0xc0015d8200)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:103 +0x1a5
      net/http.HandlerFunc.ServeHTTP(0xc0015d8100?, {0x20eebb0?, 0xc00041b058?}, 0xc000531930?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1({0x7fae682a40d8?, 0xc00041b048}, 0x9dbbaa?)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/audit.go:111 +0x549
      net/http.HandlerFunc.ServeHTTP(0xc00003def0?, {0x7fae682a40d8?, 0xc00041b048?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x7fae682a40d8, 0xc00041b048}, 0xc0015d8100)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:79 +0x178
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x7fae682a40d8?, 0xc00041b048?}, 0x0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x7fae682a40d8, 0xc00041b048}, 0xc0015d8100)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:103 +0x1a5
      net/http.HandlerFunc.ServeHTTP(0x20f0f58?, {0x7fae682a40d8?, 0xc00041b048?}, 0x20cfd00?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x7fae682a40d8, 0xc00041b048}, 0xc0015d8100)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/authentication.go:80 +0x8b9
      net/http.HandlerFunc.ServeHTTP(0x20f0f20?, {0x7fae682a40d8?, 0xc00041b048?}, 0x20cfc08?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x7fae682a40d8, 0xc00041b048}, 0xc000e69e00)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filterlatency/filterlatency.go:88 +0x46b
      net/http.HandlerFunc.ServeHTTP(0xc0019f5890?, {0x7fae682a40d8?, 0xc00041b048?}, 0xc000848764?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/server/filters.WithCORS.func1({0x7fae682a40d8, 0xc00041b048}, 0xc000e69e00)
          k8s.io/apiserver@v0.22.2/pkg/server/filters/cors.go:75 +0x10b
      net/http.HandlerFunc.ServeHTTP(0xc00149a380?, {0x7fae682a40d8?, 0xc00041b048?}, 0xc0008487d0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
          k8s.io/apiserver@v0.22.2/pkg/server/filters/timeout.go:108 +0xa2
      created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
          k8s.io/apiserver@v0.22.2/pkg/server/filters/timeout.go:94 +0x2cc
      
      goroutine 3706802 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x19eb780?, 0xc001206e20})
          k8s.io/apimachinery@v0.22.2/pkg/util/runtime/runtime.go:74 +0x99
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc0016aec60, 0x1, 0x1560f26?})
          k8s.io/apimachinery@v0.22.2/pkg/util/runtime/runtime.go:48 +0x75
      panic({0x19eb780, 0xc001206e20})
          runtime/panic.go:838 +0x207
      k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc0005047c8, {0x20eecd0?, 0xc0010fae00}, 0xdf8475800?)
          k8s.io/apiserver@v0.22.2/pkg/server/filters/timeout.go:114 +0x452
      k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x20eecd0, 0xc0010fae00}, 0xc000e69d00)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/request_deadline.go:101 +0x494
      net/http.HandlerFunc.ServeHTTP(0xc0016af048?, {0x20eecd0?, 0xc0010fae00?}, 0xc0000bc138?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1({0x20eecd0?, 0xc0010fae00}, 0xc000e69d00)
          k8s.io/apiserver@v0.22.2/pkg/server/filters/waitgroup.go:59 +0x177
      net/http.HandlerFunc.ServeHTTP(0x20f0f58?, {0x20eecd0?, 0xc0010fae00?}, 0x7fae705daff0?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.WithAuditAnnotations.func1({0x20eecd0, 0xc0010fae00}, 0xc000e69c00)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/audit_annotations.go:37 +0x230
      net/http.HandlerFunc.ServeHTTP(0x20f0f58?, {0x20eecd0?, 0xc0010fae00?}, 0x20cfc08?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x20eecd0?, 0xc0010fae00}, 0xc000e69b00)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/warning.go:35 +0x2bb
      net/http.HandlerFunc.ServeHTTP(0x1c9dda0?, {0x20eecd0?, 0xc0010fae00?}, 0xd?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x20eecd0, 0xc0010fae00}, 0x0?)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/cachecontrol.go:31 +0x126
      net/http.HandlerFunc.ServeHTTP(0x20f0f58?, {0x20eecd0?, 0xc0010fae00?}, 0x20cfc08?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/server/httplog.WithLogging.func1({0x20ef480?, 0xc001c20620}, 0xc000e69a00)
          k8s.io/apiserver@v0.22.2/pkg/server/httplog/httplog.go:103 +0x518
      net/http.HandlerFunc.ServeHTTP(0x20f0f58?, {0x20ef480?, 0xc001c20620?}, 0x20cfc08?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x20ef480, 0xc001c20620}, 0xc000e69900)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/requestinfo.go:39 +0x316
      net/http.HandlerFunc.ServeHTTP(0x20f0f58?, {0x20ef480?, 0xc001c20620?}, 0xc0007c3f70?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x20ef480, 0xc001c20620}, 0xc000e69800)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/request_received_time.go:38 +0x27e
      net/http.HandlerFunc.ServeHTTP(0x419e2c?, {0x20ef480?, 0xc001c20620?}, 0xc0007c3e40?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x20ef480?, 0xc001c20620?}, 0xc0004ff600?)
          k8s.io/apiserver@v0.22.2/pkg/server/filters/wrap.go:74 +0xb1
      net/http.HandlerFunc.ServeHTTP(0x1c05260?, {0x20ef480?, 0xc001c20620?}, 0x8?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/endpoints/filters.withAuditID.func1({0x20ef480, 0xc001c20620}, 0xc000e69600)
          k8s.io/apiserver@v0.22.2/pkg/endpoints/filters/with_auditid.go:66 +0x40d
      net/http.HandlerFunc.ServeHTTP(0x1c9dda0?, {0x20ef480?, 0xc001c20620?}, 0xd?)
          net/http/server.go:2084 +0x2f
      github.com/openshift/oauth-server/pkg/server/headers.WithPreserveAuthorizationHeader.func1({0x20ef480, 0xc001c20620}, 0xc000e69600)
          github.com/openshift/oauth-server/pkg/server/headers/oauthbasic.go:16 +0xe8
      net/http.HandlerFunc.ServeHTTP(0xc0016af9d0?, {0x20ef480?, 0xc001c20620?}, 0x16?)
          net/http/server.go:2084 +0x2f
      github.com/openshift/oauth-server/pkg/server/headers.WithStandardHeaders.func1({0x20ef480, 0xc001c20620}, 0x4d55c0?)
          github.com/openshift/oauth-server/pkg/server/headers/headers.go:30 +0x18f
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x20ef480?, 0xc001c20620?}, 0xc0016afac8?)
          net/http/server.go:2084 +0x2f
      k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc00098d622?, {0x20ef480?, 0xc001c20620?}, 0xc000401000?)
          k8s.io/apiserver@v0.22.2/pkg/server/handler.go:189 +0x2b
      net/http.serverHandler.ServeHTTP({0xc0019f5170?}, {0x20ef480, 0xc001c20620}, 0xc000e69600)
          net/http/server.go:2916 +0x43b
      net/http.(*conn).serve(0xc0002b1720, {0x20f0f58, 0xc0001e8120})
          net/http/server.go:1966 +0x5d7
      created by net/http.(*Server).Serve
          net/http/server.go:3071 +0x4db

      Version-Release number of selected component (if applicable):

      OpenShift Container Platform 4.11.13

      How reproducible:

      - Always

      Steps to Reproduce:

      1. Install OpenShift Container Platform 4.11
      2. Configure OpenID Group Sync (as per https://docs.openshift.com/container-platform/4.11/authentication/identity_providers/configuring-oidc-identity-provider.html#identity-provider-oidc-CR_configuring-oidc-identity-provider)
      3. Have users with hundrets of groups
      4. Login and after a while, remove some Groups from the user in the IDP and from OpenShift Container Platform 
      5. Try to login again and see the panic in oauth-apiserver

      Actual results:

      User is unable to login and oauth pods are reporting a panic as shown above

      Expected results:

      oauth-apiserver should invalidate the cache quickly to remove potential invalid references to non exsting groups

      Additional info:

       

            slaznick@redhat.com Stanislav Láznička
            rhn-support-sreber Simon Reber
            Zimo Xiao Zimo Xiao (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: