Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-811

Workspaces created with CRW 2.0 fail to start after update to 2.1 when OpenShift OAuth support is enabled

    XMLWordPrintable

Details

    Description

      Existing workspaces that were created in an CRW 2.0.0 installation cannot be started again after the Che server is updated to CRW 2.1.0.

      Steps to reproduce

      1. workspace created in a 2.0.0 CRW installation with default settings (it will be created in a dedicated namespace named with the workspaceId)
      2. stop the workspace
      3. Upgrade CRW to 2.1.0 (either through OperatorHub or crwctl)
      4. Start existing workspace: it will produce this error.

      Expected behavior

      Workspace should start without error inside its original namespace

      Installation method

      Either with crwctl or OperatorHub

      che config map of the 2.0.0 Che server install with default settings + TLS (=> Openshift OAuth + TLS + selfsigned certs):

        CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: 'true'
        CHE_PORT: '8080'
        CHE_INFRA_KUBERNETES_PVC_STRATEGY: per-workspace
        CHE_JDBC_PASSWORD: hDw1kyi6nke2
        CHE_KEYCLOAK_REALM: codeready
        CHE_INFRA_KUBERNETES_TRUST__CERTS: 'true'
        CHE_INFRASTRUCTURE_ACTIVE: openshift
        CHE_WORKSPACE_NO__PROXY: ''
        CHE_WORKSPACE_PLUGIN__BROKER_UNIFIED_IMAGE: 'registry.redhat.io/codeready-workspaces/pluginbroker-rhel8:2.0'
        CHE_INFRA_OPENSHIFT_PROJECT: ''
        CHE_MULTIUSER: 'true'
        CHE_HOST: codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com
        CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: 'registry.redhat.io/codeready-workspaces/jwtproxy-rhel8:2.0'
        CHE_WEBSOCKET_ENDPOINT__MINOR: >-
          wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket-minor
        CHE_LOG_LEVEL: INFO
        CHE_JDBC_URL: 'jdbc:postgresql://postgres:5432/dbche'
        CHE_WORKSPACE_DEVFILE__REGISTRY__URL: >-
          https://devfile-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com
        CHE_WEBSOCKET_ENDPOINT: >-
          wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket
        CHE_WORKSPACE_HTTPS__PROXY: ''
        CHE_DEBUG_SERVER: 'false'
        CHE_INFRA_KUBERNETES_SERVICE__ACCOUNT__NAME: che-workspace
        CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER: openshift-v4
        CHE_WORKSPACE_JAVA__OPTIONS: >-
          -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
          -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
          -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
          -Djava.security.egd=file:/dev/./urandom 
        CHE_KEYCLOAK_CLIENT__ID: codeready-public
        CHE_INFRA_KUBERNETES_PVC_STORAGE__CLASS__NAME: ''
        CHE_KEYCLOAK_AUTH__SERVER__URL: 'https://keycloak-test-david-prod-update.apps.test-ocp43.codereadyqe.com/auth'
        CHE_WORKSPACE_PLUGIN__REGISTRY__URL: >-
          https://plugin-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com/v3
        CHE_WORKSPACE_PLUGIN__BROKER_INIT_IMAGE: 'registry.redhat.io/codeready-workspaces/pluginbrokerinit-rhel8:2.0'
        CHE_INFRA_KUBERNETES_PVC_QUANTITY: 1Gi
        CHE_JDBC_USERNAME: pgche
        CHE_INFRA_OPENSHIFT_TLS__ENABLED: 'true'
        KUBERNETES_LABEL: 'app=codeready,component=codeready'
        CHE_API: 'https://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api'
        CHE_WORKSPACE_MAVEN__OPTIONS: >-
          -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
          -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
          -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
          -Djava.security.egd=file:/dev/./urandom 
        JAVA_OPTS: >-
          -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
          -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
          -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
          -Dsun.zip.disableMemoryMapping=true -Xms20m 
        CHE_WORKSPACE_HTTP__PROXY: ''
        CHE_INFRA_KUBERNETES_PVC_JOBS_IMAGE: 'registry.redhat.io/ubi8-minimal:8.0-213'
        CHE_WORKSPACE_HTTP__PROXY__JAVA__OPTIONS: ''
      
      

      Che server POD env variables

          env:
              - name: CM_REVISION
                value: '4727172'
              - name: KUBERNETES_NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
              - name: CHE_SELF__SIGNED__CERT
                valueFrom:
                  secretKeyRef:
                    name: self-signed-certificate
                    key: ca.crt
                    optional: true
      
      

      Now Config map of the 2.1.0 Che server:

        CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: 'true'
        CHE_PORT: '8080'
        CHE_INFRA_KUBERNETES_PVC_STRATEGY: per-workspace
        CHE_JDBC_PASSWORD: hDw1kyi6nke2
        CHE_KEYCLOAK_REALM: codeready
        CHE_INFRA_KUBERNETES_TRUST__CERTS: 'true'
        CHE_INFRASTRUCTURE_ACTIVE: openshift
        CHE_WORKSPACE_NO__PROXY: ''
        CHE_INFRA_KUBERNETES_NAMESPACE_ALLOW__USER__DEFINED: 'false'
        CHE_MULTIUSER: 'true'
        CHE_METRICS_ENABLED: 'false'
        CHE_HOST: codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com
        CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: >-
          registry.redhat.io/codeready-workspaces/jwtproxy-rhel8@sha256:63182dae6377ac01fbebdcb9c6a4435681ea88521cfc6e4222fe3920a3641127
        CHE_WEBSOCKET_ENDPOINT__MINOR: >-
          wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket-minor
        CHE_LOG_LEVEL: INFO
        CHE_JDBC_URL: 'jdbc:postgresql://postgres:5432/dbche'
        CHE_WORKSPACE_DEVFILE__REGISTRY__URL: >-
          https://devfile-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com
        CHE_WEBSOCKET_ENDPOINT: >-
          wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket
        CHE_WORKSPACE_HTTPS__PROXY: ''
        CHE_DEBUG_SERVER: 'false'
        CHE_INFRA_KUBERNETES_SERVICE__ACCOUNT__NAME: che-workspace
        CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER: openshift-v4
        CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: <username>-codeready
        CHE_WORKSPACE_JAVA__OPTIONS: >-
          -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
          -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
          -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
          -Djava.security.egd=file:/dev/./urandom 
        CHE_WORKSPACE_PLUGIN__BROKER_ARTIFACTS_IMAGE: >-
          registry.redhat.io/codeready-workspaces/pluginbroker-artifacts-rhel8@sha256:5815bab69fc343cbf6dac0fd67dd70a25757fac08689a15e4a762655fa2e8a2c
        CHE_KEYCLOAK_CLIENT__ID: codeready-public
        CHE_WORKSPACE_PLUGIN__BROKER_METADATA_IMAGE: >-
          registry.redhat.io/codeready-workspaces/pluginbroker-metadata-rhel8@sha256:6c9abe63a70a6146dc49845f2f7732e3e6e0bcae6a19c3a6557367d6965bc1f8
        CHE_INFRA_KUBERNETES_PVC_STORAGE__CLASS__NAME: ''
        CHE_KEYCLOAK_AUTH__SERVER__URL: 'https://keycloak-test-david-prod-update.apps.test-ocp43.codereadyqe.com/auth'
        CHE_WORKSPACE_PLUGIN__REGISTRY__URL: >-
          https://plugin-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com/v3
        CHE_INFRA_KUBERNETES_PVC_QUANTITY: 1Gi
        KUBERNETES_LABELS: 'app=codeready,component=codeready'
        CHE_JDBC_USERNAME: pgche
        CHE_INFRA_OPENSHIFT_TLS__ENABLED: 'true'
        CHE_API: 'https://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api'
        CHE_WORKSPACE_MAVEN__OPTIONS: >-
          -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
          -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
          -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
          -Djava.security.egd=file:/dev/./urandom 
        JAVA_OPTS: >-
          -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
          -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
          -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
          -Dsun.zip.disableMemoryMapping=true -Xms20m 
        CHE_WORKSPACE_HTTP__PROXY: ''
        CHE_INFRA_KUBERNETES_PVC_JOBS_IMAGE: >-
          registry.access.redhat.com/ubi8-minimal@sha256:9285da611437622492f9ef4229877efe302589f1401bbd4052e9bb261b3d4387
        CHE_WORKSPACE_HTTP__PROXY__JAVA__OPTIONS: ''
      
      

      And additional variables of the new 2.1.0 Che server:

              - name: CM_REVISION
                value: '4732509'
              - name: KUBERNETES_NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
              - name: CHE_SELF__SIGNED__CERT
                valueFrom:
                  secretKeyRef:
                    name: self-signed-certificate
                    key: ca.crt
                    optional: true
              - name: CHE_GIT_SELF__SIGNED__CERT
              - name: CHE_GIT_SELF__SIGNED__CERT__HOST
      
      

      Che server logs

      15-Apr-2020 12:51:47.844 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 13244 ms
      2020-04-15 12:53:53,027[nio-8080-exec-4]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 494]   - Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspace0fyulpv8u0w7ne3s. Message: Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspace0fyulpv8u0w7ne3s, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
      org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInfrastructureException: Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspace0fyulpv8u0w7ne3s. Message: Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspace0fyulpv8u0w7ne3s, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
      	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.update(OpenShiftProject.java:193)
      	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.prepare(OpenShiftProject.java:120)
      	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProjectFactory.getOrCreate(OpenShiftProjectFactory.java:86)
      	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:69)
      	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:31)
      	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:466)
      	at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:488)
      	at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:373)
      	at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:132)
      	at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:458)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
      	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
      	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
      	at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297)
      	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)
      	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
      	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
      	at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
      	at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
      	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
      	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
      	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
      	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
      	at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
      	at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:142)
      	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
      	at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
      	at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
      	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
      	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
      	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
      	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
      	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspace0fyulpv8u0w7ne3s. Message: Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspace0fyulpv8u0w7ne3s, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:476)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:415)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:261)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:243)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:785)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:109)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:104)
      	at io.fabric8.openshift.api.model.DoneableProject.done(DoneableProject.java:27)
      	at io.fabric8.openshift.api.model.DoneableProject.done(DoneableProject.java:6)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:116)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:37)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:420)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:415)
      	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.update(OpenShiftProject.java:186)
      	... 58 common frames omitted
      
      

      Attachments

        Issue Links

          Activity

            People

              skabashn Sergii Kabashniuk
              dfestal David Festal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: