Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-1920

CRW 2.9.0 workspace failed to start because was unable to attach or mount volumes for default token

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • 2.10.0.GA
    • 2.9.0.GA
    • testing
    • False
    • False
    • Hide
      = Workspace failing to start with the `common` or `unique` storage strategy

      When using the `common` or `unique` storage strategy, {prod-short} server may retrieve a stale token from {identity-provider} during workspace startup. OpenShift events displays the error "Unable to attach or mount volumes". The workspace fails to start displaying the error "Your session has expired".

      .Workaround

      Use the `per-workspace` storage strategy, or change access mode of persistent volume to "ReadWriteMany". See link:{prod-ig-url}configuring-storage-strategies_crw[Configuring storage strategies].
      Show
      = Workspace failing to start with the `common` or `unique` storage strategy When using the `common` or `unique` storage strategy, {prod-short} server may retrieve a stale token from {identity-provider} during workspace startup. OpenShift events displays the error "Unable to attach or mount volumes". The workspace fails to start displaying the error "Your session has expired". .Workaround Use the `per-workspace` storage strategy, or change access mode of persistent volume to "ReadWriteMany". See link:{prod-ig-url}configuring-storage-strategies_crw[Configuring storage strategies].
    • Workaround Exists
    • Hide

      Set "pvcStrategy: per-workspace", or change access mode of persistent volume to "ReadWriteMany".

      https://access.redhat.com/documentation/en-us/red_hat_codeready_workspaces/2.8/html-single/installation_guide/index#configuring-storage-strategies_crw

      https://access.redhat.com/documentation/en-us/red_hat_codeready_workspaces/1.0/html/administration_guide/codeready-workspaces_administration_guide#how-the-che-server-uses-PVCs-and-PVs-for-storage

      Show
      Set "pvcStrategy: per-workspace", or change access mode of persistent volume to "ReadWriteMany". https://access.redhat.com/documentation/en-us/red_hat_codeready_workspaces/2.8/html-single/installation_guide/index#configuring-storage-strategies_crw https://access.redhat.com/documentation/en-us/red_hat_codeready_workspaces/1.0/html/administration_guide/codeready-workspaces_administration_guide#how-the-che-server-uses-PVCs-and-PVs-for-storage

    Description

      CRW 2.9.0.RC-06-11 workspace with id "workspace4gsntnb7xkvuj7ge" failed to start on AWS because of problem with volume mount:





       
      CRW instance: https://codeready-crw-crwctl.apps.ocp47-aws.crw-qe.com/
      CRW project: https://console-openshift-console.apps.ocp47.crw-qe.com/k8s/cluster/projects/crw-crwctl

      error message
      Unable to attach or mount volumes: unmounted volumes=[claim-che-workspace], unattached volumes=[default-token-9gn7q claim-che-workspace]: timed out waiting for the condition
      server logs
      2021-06-14 09:47:33,826[nio-8080-exec-2]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 694]    - Workspace 'admin/java-eap-maven-6u8v4' with id 'workspace4gsntnb7xkvuj7ge' created by user 'admin'
2021-06-14 09:47:35,979[nio-8080-exec-5]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 480]   - Starting workspace 'admin/java-eap-maven-6u8v4' with id 'workspace4gsntnb7xkvuj7ge' by user 'admin'
2021-06-14 09:49:31,281[nio-8080-exec-1]  [ERROR] [.m.m.a.s.s.SignatureKeyManager 141]  - Unable to generate signature keypair for ws workspacepei1mxkprz3hiwbf. Cause: Unable to create signature key pair because referenced workspace with id 'workspacepei1mxkprz3hiwbf' doesn't exist
2021-06-14 09:49:31,281[nio-8080-exec-1]  [ERROR] [.m.m.a.s.s.SignatureKeyManager 105]  - Failed to load signature keys for ws  workspacepei1mxkprz3hiwbf. Cause: Unable to create signature key pair because referenced workspace with id 'workspacepei1mxkprz3hiwbf' doesn't exist
2021-06-14 09:52:08,628[er-ThreadPool-0]  [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223]  - Unable to perform '[rm, -rf, /tmp/job_mount/workspace33awnbzfy6oisxmh]' command for the workspace 'workspace33awnbzfy6oisxmh' cause: 'Waiting for pod 'rm-workspace33awnbzfy6oisxmh' reached timeout'
2021-06-14 09:52:08,636[er-ThreadPool-0]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:841)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:233)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:191)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.lambda$removeDirsAsync$0(PVCSubPathHelper.java:155)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:08,909[er-ThreadPool-1]  [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223]  - Unable to perform '[rm, -rf, /tmp/job_mount/workspaceyjk8pbxy6t3gwojl]' command for the workspace 'workspaceyjk8pbxy6t3gwojl' cause: 'Waiting for pod 'rm-workspaceyjk8pbxy6t3gwojl' reached timeout'
2021-06-14 09:52:08,915[er-ThreadPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:841)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:233)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:191)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.lambda$removeDirsAsync$0(PVCSubPathHelper.java:155)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:08,921[er-ThreadPool-2]  [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223]  - Unable to perform '[rm, -rf, /tmp/job_mount/workspacebrt14qoo8im0eukv]' command for the workspace 'workspacebrt14qoo8im0eukv' cause: 'Waiting for pod 'rm-workspacebrt14qoo8im0eukv' reached timeout'
2021-06-14 09:52:08,924[er-ThreadPool-2]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:841)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:233)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:191)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.lambda$removeDirsAsync$0(PVCSubPathHelper.java:155)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:08,991[er-ThreadPool-3]  [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223]  - Unable to perform '[rm, -rf, /tmp/job_mount/workspacez3r4uav30hz52bzb]' command for the workspace 'workspacez3r4uav30hz52bzb' cause: 'Waiting for pod 'rm-workspacez3r4uav30hz52bzb' reached timeout'
2021-06-14 09:52:08,995[er-ThreadPool-3]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:841)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:233)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:191)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.lambda$removeDirsAsync$0(PVCSubPathHelper.java:155)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,674[aceSharedPool-1]  [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223]  - Unable to perform '[mkdir, -m, 777, -p, /tmp/job_mount/workspace4gsntnb7xkvuj7ge/che-logs-che-plugin-broker/]' command for the workspace 'workspace4gsntnb7xkvuj7ge' cause: 'Waiting for pod 'mkdir-workspace4gsntnb7xkvuj7ge' reached timeout'
2021-06-14 09:52:36,681[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:841)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:233)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.execute(PVCSubPathHelper.java:167)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.PVCSubPathHelper.createDirs(PVCSubPathHelper.java:136)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.pvc.CommonPVCStrategy.prepare(CommonPVCStrategy.java:221)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:67)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,683[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesConfigsMaps.create(KubernetesConfigsMaps.java:70)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:98)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,686[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:875)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:140)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,686[aceSharedPool-1]  [ERROR] [o.e.c.w.i.k.w.b.DeployBroker 142]    - Brokers pod removal failed. Error: Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:198)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:875)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:140)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,688[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesSecrets.delete(KubernetesSecrets.java:86)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:145)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,688[aceSharedPool-1]  [ERROR] [o.e.c.w.i.k.w.b.DeployBroker 147]    - Brokers secret removal failed. Error: Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:198)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesSecrets.delete(KubernetesSecrets.java:86)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:145)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,690[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesConfigsMaps.delete(KubernetesConfigsMaps.java:87)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:150)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,691[aceSharedPool-1]  [ERROR] [o.e.c.w.i.k.w.b.DeployBroker 152]    - Brokers config map removal failed. Error: Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:198)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesConfigsMaps.delete(KubernetesConfigsMaps.java:87)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:150)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,691[aceSharedPool-1]  [WARN ] [.i.k.KubernetesInternalRuntime 257]  - Failed to start Kubernetes runtime of workspace workspace4gsntnb7xkvuj7ge.
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:198)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesConfigsMaps.create(KubernetesConfigsMaps.java:70)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:98)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129)
	at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,693[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.createOC(OpenShiftClientFactory.java:97)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftRoutes.delete(OpenShiftRoutes.java:84)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespace.doRemove(KubernetesNamespace.java:288)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.cleanUp(OpenShiftProject.java:157)
	at org.eclipse.che.workspace.infrastructure.kubernetes.RuntimeCleaner.cleanUp(RuntimeCleaner.java:56)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:266)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,695[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesServices.delete(KubernetesServices.java:86)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespace.doRemove(KubernetesNamespace.java:288)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.cleanUp(OpenShiftProject.java:157)
	at org.eclipse.che.workspace.infrastructure.kubernetes.RuntimeCleaner.cleanUp(RuntimeCleaner.java:56)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:266)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,698[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments.delete(KubernetesDeployments.java:875)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespace.doRemove(KubernetesNamespace.java:288)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.cleanUp(OpenShiftProject.java:157)
	at org.eclipse.che.workspace.infrastructure.kubernetes.RuntimeCleaner.cleanUp(RuntimeCleaner.java:56)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:266)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,700[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesSecrets.delete(KubernetesSecrets.java:86)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespace.doRemove(KubernetesNamespace.java:288)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.cleanUp(OpenShiftProject.java:157)
	at org.eclipse.che.workspace.infrastructure.kubernetes.RuntimeCleaner.cleanUp(RuntimeCleaner.java:56)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:266)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,702[aceSharedPool-1]  [ERROR] [.IdentityProviderConfigFactory 192]  - Cannot retrieve User OpenShift token from the 'openshift-v4' identity provider
org.eclipse.che.api.core.BadRequestException: Invalid token.
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.doRequest(KeycloakServiceClient.java:199)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakServiceClient.getIdentityProviderToken(KeycloakServiceClient.java:137)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.personalizeConfig(IdentityProviderConfigFactory.java:175)
	at org.eclipse.che.workspace.infrastructure.openshift.multiuser.oauth.IdentityProviderConfigFactory.buildConfig(IdentityProviderConfigFactory.java:169)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftClientFactory.buildConfig(OpenShiftClientFactory.java:152)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesClientFactory.create(KubernetesClientFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesConfigsMaps.delete(KubernetesConfigsMaps.java:87)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespace.doRemove(KubernetesNamespace.java:288)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.cleanUp(OpenShiftProject.java:157)
	at org.eclipse.che.workspace.infrastructure.kubernetes.RuntimeCleaner.cleanUp(RuntimeCleaner.java:56)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:266)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,703[aceSharedPool-1]  [WARN ] [.i.k.KubernetesInternalRuntime 268]  - Failed to clean up namespace after workspace 'workspace4gsntnb7xkvuj7ge' start failing.
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Error(s) occurs while cleaning up the namespace. Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespace.doRemove(KubernetesNamespace.java:302)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.cleanUp(OpenShiftProject.java:157)
	at org.eclipse.che.workspace.infrastructure.kubernetes.RuntimeCleaner.cleanUp(RuntimeCleaner.java:56)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:266)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:75)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-14 09:52:36,743[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 995]   - Workspace 'admin:java-eap-maven-6u8v4' with id 'workspace4gsntnb7xkvuj7ge' start failed

      PVC options:

      pvcStrategy: common
pvcClaimSize: 2Gi

      PV options:

      • "Access Modes: ReadWriteOnce"

      Attachments

        Activity

          People

            dnochevn Dmytro Nochevnov
            dnochevn Dmytro Nochevnov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: