Uploaded image for project: 'Arquillian'
  1. Arquillian
  2. ARQ-1282

TestNG @DataProvider is fully invoked for each record

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.3.Final
    • Fix Version/s: None
    • Component/s: JBoss AS Containers
    • Labels:
      None
    • Steps to Reproduce:
      Hide

      TestCase Class

      DataProviderTest.java
      import org.jboss.arquillian.container.test.api.Deployment;
      import org.jboss.arquillian.container.test.api.OverProtocol;
      import org.jboss.arquillian.testng.Arquillian;
      import org.jboss.shrinkwrap.api.Archive;
      import org.jboss.shrinkwrap.api.ShrinkWrap;
      import org.jboss.shrinkwrap.api.asset.EmptyAsset;
      import org.jboss.shrinkwrap.api.spec.WebArchive;
      import org.testng.annotations.DataProvider;
      import org.testng.annotations.Test;
      
      import static org.testng.Assert.*;
      
      public class DataProviderTest extends Arquillian {
      
          @Deployment
          @OverProtocol( "Servlet 3.0" )
          public static Archive<?> getDeployment() {
              return ShrinkWrap.create( WebArchive.class ).addAsWebInfResource( EmptyAsset.INSTANCE, "beans.xml" );
          }
      
          @Test( dataProvider = "sumProvider" )
          public void testSum( int a, int b, int expectedSum ) {
      
              // execute test
              int result = a + b;
      
              // log it and watch wrong behaviour
              // HERE IS A BUG !!!
              // Look into server log to NUMBER of logged events.
              System.out.println( String.format( "Execution of sum: %1$d + %2$d = %3$d", a, b, result ) );
      
              // verify result
              assertEquals( result, expectedSum );
          }
      
          @DataProvider
          public Object[][] sumProvider() {
              return new Object[][]{
                      new Object[]{ 0, 0, 0 },
                      new Object[]{ 1, 1, 2 },
                      new Object[]{ 1, 5, 6 },
                      new Object[]{ 5, 1, 6 }
              };
          }
      
      }
      
      Show
      TestCase Class DataProviderTest.java import org.jboss.arquillian.container.test.api.Deployment; import org.jboss.arquillian.container.test.api.OverProtocol; import org.jboss.arquillian.testng.Arquillian; import org.jboss.shrinkwrap.api.Archive; import org.jboss.shrinkwrap.api.ShrinkWrap; import org.jboss.shrinkwrap.api.asset.EmptyAsset; import org.jboss.shrinkwrap.api.spec.WebArchive; import org.testng.annotations.DataProvider; import org.testng.annotations.Test; import static org.testng.Assert.*; public class DataProviderTest extends Arquillian { @Deployment @OverProtocol( "Servlet 3.0" ) public static Archive<?> getDeployment() { return ShrinkWrap.create( WebArchive.class ).addAsWebInfResource( EmptyAsset.INSTANCE, "beans.xml" ); } @Test( dataProvider = "sumProvider" ) public void testSum( int a, int b, int expectedSum ) { // execute test int result = a + b; // log it and watch wrong behaviour // HERE IS A BUG !!! // Look into server log to NUMBER of logged events. System .out.println( String .format( "Execution of sum: %1$d + %2$d = %3$d" , a, b, result ) ); // verify result assertEquals( result, expectedSum ); } @DataProvider public Object [][] sumProvider() { return new Object [][]{ new Object []{ 0, 0, 0 }, new Object []{ 1, 1, 2 }, new Object []{ 1, 5, 6 }, new Object []{ 5, 1, 6 } }; } }

      Description

      Expected behavior

      TestNG defines @DataProvider annotation to allow single method to perform multiple test cases. Provider method annotated with @DataProvider returns an array of arrays, each containing one test case. These parameters are given to the target method as input parameters. By the definition, the total count of performed test by single method is equal to number of test cases in related data provider.

      Actual behavior

      Using TestNG under Arquillian framework makes this functionality buggy. Although the client (maven, IDE, etc.) thinks, that the total number of invoked test is correct, it is not. When we take a look into server's log, we can see that for each single test case it invoked full set of all test cases. It means, that in the end the amount of performed tests is equal to expected count squared. Such behaviour is not only slow but also in some cases it doesn't work and it makes tests to fail.

      Example

      @DataProvider
      public Object[][] sumProvider() {
          return new Object[][]{
              new Object[]{ 0, 0, 0 },
              new Object[]{ 1, 1, 2 },
              new Object[]{ 1, 5, 6 },
              new Object[]{ 5, 1, 6 }
          };
      }
      

      Expected output in server log

      Execution of sum: 0 + 0 = 0
      Execution of sum: 1 + 1 = 2
      Execution of sum: 1 + 5 = 6
      Execution of sum: 5 + 1 = 6

      Actual output

      Execution of sum: 0 + 0 = 0
      Execution of sum: 1 + 1 = 2
      Execution of sum: 1 + 5 = 6
      Execution of sum: 5 + 1 = 6
      Execution of sum: 0 + 0 = 0
      Execution of sum: 1 + 1 = 2
      Execution of sum: 1 + 5 = 6
      Execution of sum: 5 + 1 = 6
      Execution of sum: 0 + 0 = 0
      Execution of sum: 1 + 1 = 2
      Execution of sum: 1 + 5 = 6
      Execution of sum: 5 + 1 = 6
      Execution of sum: 0 + 0 = 0
      Execution of sum: 1 + 1 = 2
      Execution of sum: 1 + 5 = 6
      Execution of sum: 5 + 1 = 6

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                bmajsak Bartosz Majsak
                Reporter:
                karel.cemus Karel Cemus
              • Votes:
                15 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated: