Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-8643

[OCP 3.6] f-m-p stuck in waitUntilBuildFinished

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • FIS 2.x-GA
    • FIS 2.0
    • FIS-Fabric8
    • None
    • % %
    • FIS 2.0 R4

    Description

      Running builds via f-m-p on current OCP 3.6 builds make f-m-p stuck waiting for the build to finish, even when the build has already finished

      "main" #1 prio=5 os_prio=0 tid=0x00007f016c00a000 nid=0x3fb3 waiting on condition [0x00007f0173372000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000794a7e8a0> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
      	at io.fabric8.maven.core.openshift.BuildService.waitUntilBuildFinished(BuildService.java:99)
      	at io.fabric8.maven.core.openshift.BuildService.waitForOpenShiftBuildToComplete(BuildService.java:84)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.executeOpenShiftBuild(BuildMojo.java:314)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.buildAndTag(BuildMojo.java:237)
      	at io.fabric8.maven.docker.BuildMojo.executeInternal(BuildMojo.java:47)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.executeInternal(BuildMojo.java:218)
      	at io.fabric8.maven.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:193)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.execute(BuildMojo.java:201)
      	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:352)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:197)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
      	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
      	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
      	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
      	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
      	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
      	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
      	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
      	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
      	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
      
      

      Comparing behavior with OCP 3.5 and OCP 3.6 suggests that there may be a race in which the websocket watcher is started after the build has ended, thus missing the build completition event... (or perhaps the new ocp version does not allow that many connections at once, thus blocking the watcher websocket connection until the build log watcher ends?)

      OCP 3.6:

      [INFO] F8: Starting Build spring-boot-camel-amq-s2i
      [INFO] F8: Waiting for build spring-boot-camel-amq-s2i-1 to complete...
      [INFO] F8: Receiving source from STDIN as archive ...
      [INFO] F8: ==================================================================
      [INFO] F8: Starting S2I Java Build .....
      [INFO] F8: S2I binary build from fabric8-maven-plugin detected
      [INFO] F8: Copying binaries from /tmp/src/maven to /deployments ...
      [INFO] F8: ... done
      [INFO] F8: 
      [INFO] F8: Pushing image 172.30.158.237:5000/maschmid/spring-boot-camel-amq:latest ...
      [INFO] F8: Pushed 5/6 layers, 84% complete
      [INFO] F8: Pushed 6/6 layers, 100% complete
      [INFO] F8: Push successful
      [DEBUG] WebSocket successfully opened
      

      ... and hangs

      OCP 3.5:

      [INFO] F8: Starting Build spring-boot-camel-amq-s2i
      [INFO] F8: Waiting for build spring-boot-camel-amq-s2i-1 to complete...
      [INFO] F8: Receiving source from STDIN as archive ...
      [DEBUG] WebSocket successfully opened
      [INFO] F8: ==================================================================
      [INFO] F8: Starting S2I Java Build .....
      [INFO] F8: S2I binary build from fabric8-maven-plugin detected
      [INFO] F8: Copying binaries from /tmp/src/maven to /deployments ...
      [INFO] F8: ... done
      [INFO] F8: 
      [INFO] F8: Pushing image 172.30.45.210:5000/maschmid/spring-boot-camel-amq:latest ...
      [INFO] F8: Pushed 4/6 layers, 67% complete
      [INFO] F8: Pushed 5/6 layers, 97% complete
      [INFO] F8: Pushed 6/6 layers, 100% complete
      [INFO] F8: Push successful
      [INFO] F8: Build spring-boot-camel-amq-s2i-1 Complete
      [INFO] F8: Found tag on ImageStream spring-boot-camel-amq tag: sha256:8a665c4d0c0c77318b3051d8f861de26c8ee16dbba1fa3c0372ff63023d845d5
      [INFO] F8: Imagestream spring-boot-camel-amq written to /home/maschmid/devel/xpaas-qe/master/test-fuse/tmp/spring-boot-camel-amq1415973405485968931/spring-boot-camel-amq/target/spring-boot-camel-amq-is.yml
      [INFO] 
      [INFO] --- maven-install-plugin:2.4:install (default-install) @ spring-boot-camel-amq ---
      ...
      

      ..and continues...

      Attachments

        1. log-3.6.txt
          434 kB
        2. log-3.5.txt
          446 kB
        3. jstack.txt
          24 kB

        Activity

          People

            Unassigned Unassigned
            maschmid@redhat.com Marek Schmidt
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: