Uploaded image for project: 'WildFly WIP'
  1. WildFly WIP
  2. WFWIP-437

Error running IIOP test module with remote EC2 nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Blocker
    • IIOP
    • None
    • Hide

      create two EC2 nodes with no networking restrictions e.g.

      aws ec2 run-instances --image-id ami-00753c9d2b9d4273c --count 2 --instance-type t3.medium --key-name jclouds-jenkins --security-groups allow-any

      note down for each instance:

      • Public IPv4 address: used later to configure the server
      • Private IPv4 address: used later to configure the test-suite

      configure the two nodes to run with the standalone-ec2-full-ha.xml profile (see Launching Clustered JBoss EAP), e.g.:

      ACCESS_KEY_ID='...'
SECRET_ACCESS_KEY='...'
S3_PING_BUCKET='...'
NODE_NAME='node0' # for the first node and 'node1' for the second node
EAP_PROFILE=standalone-ec2-full-ha.xml
JBOSS_HOME=/opt/rh/eap7/root/usr/share/wildfly
INTERNAL_IP_ADDRESS=`ip addr show | grep eth0 -A 2 | head -n 3 | tail -n 1 | awk '{ print $2 }' | sed "s-/24--g" | cut -d'/' -f1`
SERVICE_CONF_FILE=/etc/opt/rh/eap7/wildfly/eap7-standalone.conf

# Configure JBoss EAP to use the ec2-ha profile
cp /opt/rh/eap7/root/usr/share/wildfly/docs/examples/configs/$EAP_PROFILE $JBOSS_HOME/standalone/configuration/$EAP_PROFILE
chown jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE
chgrp jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE
chmod --reference=$JBOSS_HOME/standalone/configuration/standalone.xml $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# add managent user
./add-user.sh --user admin --password admin

# node name config
sudo sed -i 's/\(.*subsystem xmlns="urn:jboss:domain:undertow[^"]*"\)\(.*\)/\1 instance-id="${jboss.jvmRoute}"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# use tcp stack for clustering (using udp the cluster doens't get constituited)
sudo sed -i 's/\(.*channel name\="ee" stack\=\)"udp"\(.*\)/\1"tcp"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# https://access.redhat.com/solutions/2759881
sudo sed -i 's/\(.*\)<cluster password="[^"]*"\/>\(.*\)/\1<cluster password="admin" user="admin123"\/>\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# standalone-ec2-full-ha.xml
echo "WILDFLY_SERVER_CONFIG=$EAP_PROFILE" >> $SERVICE_CONF_FILE
echo "WILDFLY_BIND=$INTERNAL_IP_ADDRESS" >> $SERVICE_CONF_FILE

# node0
echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf

# node1 needs to have a port-offset=100 according to arquillian.xml file
echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.socket.binding.port-offset=100 -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf

echo "" >> $JBOSS_HOME/bin/standalone.conf

systemctl enable eap7-standalone
systemctl start eap7-standalone

      configure the AS test-suite to work with running EAP instances;

      e.g. in jboss-eap-7.4-src/testsuite/integration/iiop/src/test/config/arq/arquillian.xml add the following on every configured node:

      <property name="allowConnectingToRunningServer">true</property>
<property name="username">admin</property>
<property name="password">admin</property>

      Also the following property needs to be adjusted in order to pint to the JBoss EAP installation shipped with the AMI

      <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=${basedir}/target/jbossas-iiop-client</property>

      <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=/opt/rh/eap7/root/usr/share/wildfly</property>

      build the project

      e.g.

      ./build.sh clean install -DskipTests -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository -Denforcer.skip=true
./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true

      run the IIOP test module (replace the IPs with the actual IPs of your EC2 nodes)

      e.g.

      ./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true -Dnode0=3.85.211.95 -Dnode1=3.223.141.15 -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository
      Show
      create two EC2 nodes with no networking restrictions e.g. aws ec2 run-instances --image-id ami-00753c9d2b9d4273c --count 2 --instance-type t3.medium --key-name jclouds-jenkins --security-groups allow-any note down for each instance: Public IPv4 address: used later to configure the server Private IPv4 address: used later to configure the test-suite configure the two nodes to run with the standalone-ec2-full-ha.xml profile (see Launching Clustered JBoss EAP ), e.g.: ACCESS_KEY_ID='...' SECRET_ACCESS_KEY='...' S3_PING_BUCKET='...' NODE_NAME='node0' # for the first node and 'node1' for the second node EAP_PROFILE=standalone-ec2-full-ha.xml JBOSS_HOME=/opt/rh/eap7/root/usr/share/wildfly INTERNAL_IP_ADDRESS=`ip addr show | grep eth0 -A 2 | head -n 3 | tail -n 1 | awk '{ print $2 }' | sed "s-/24--g" | cut -d'/' -f1` SERVICE_CONF_FILE=/etc/opt/rh/eap7/wildfly/eap7-standalone.conf # Configure JBoss EAP to use the ec2-ha profile cp /opt/rh/eap7/root/usr/share/wildfly/docs/examples/configs/$EAP_PROFILE $JBOSS_HOME/standalone/configuration/$EAP_PROFILE chown jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE chgrp jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE chmod --reference=$JBOSS_HOME/standalone/configuration/standalone.xml $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # add managent user ./add-user.sh --user admin --password admin # node name config sudo sed -i 's/\(.*subsystem xmlns="urn:jboss:domain:undertow[^"]*"\)\(.*\)/\1 instance-id="${jboss.jvmRoute}"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # use tcp stack for clustering (using udp the cluster doens't get constituited) sudo sed -i 's/\(.*channel name\="ee" stack\=\)"udp"\(.*\)/\1"tcp"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # https://access.redhat.com/solutions/2759881 sudo sed -i 's/\(.*\)<cluster password="[^"]*"\/>\(.*\)/\1<cluster password="admin" user="admin123"\/>\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # standalone-ec2-full-ha.xml echo "WILDFLY_SERVER_CONFIG=$EAP_PROFILE" >> $SERVICE_CONF_FILE echo "WILDFLY_BIND=$INTERNAL_IP_ADDRESS" >> $SERVICE_CONF_FILE # node0 echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf # node1 needs to have a port-offset=100 according to arquillian.xml file echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.socket.binding.port-offset=100 -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf echo "" >> $JBOSS_HOME/bin/standalone.conf systemctl enable eap7-standalone systemctl start eap7-standalone configure the AS test-suite to work with running EAP instances; e.g. in jboss-eap-7.4-src/testsuite/integration/iiop/src/test/config/arq/arquillian.xml add the following on every configured node: <property name="allowConnectingToRunningServer">true</property> <property name="username">admin</property> <property name="password">admin</property> Also the following property needs to be adjusted in order to pint to the JBoss EAP installation shipped with the AMI <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=${basedir}/target/jbossas-iiop-client</property> <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=/opt/rh/eap7/root/usr/share/wildfly</property> build the project e.g. ./build.sh clean install -DskipTests -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository -Denforcer.skip=true ./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true run the IIOP test module (replace the IPs with the actual IPs of your EC2 nodes) e.g. ./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true -Dnode0=3.85.211.95 -Dnode1=3.223.141.15 -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository

    Description

      We run the IIOP test module in the following scenario:

      • we created two EC2 nodes using AMI with name RHEL-7-JBEAP-7.4.0_HVM_GA-20210909-x86_64-0-Access2-GP2
      • on a node external to AWS (e.g. a Fedora laptop) we run the IIOP AS test-suite module

      The test log shows the following error (complete log in attached file test.log): 

      [INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase
[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 17.489 s <<< FAILURE! - in org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase
[ERROR] testHomeHandle(org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase)  Time elapsed: 1.437 s  <<< ERROR!
javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:268)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:390)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:160)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	...
Caused by: javax.naming.NamingException: WFLYNAM0062: Failed to lookup env/home [Root exception is java.lang.RuntimeException: javax.naming.NamingException: WFLYIIOP0031: Cannot connect to ORB [Root exception is org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No]]
	at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:159)
	at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:83)
	at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:207)
	at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:193)
	at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:189)
	at org.jboss.as.naming.deployment.ContextNames$BindInfo$1$1.getReference(ContextNames.java:322)
	... 216 more
Caused by: java.lang.RuntimeException: javax.naming.NamingException: WFLYIIOP0031: Cannot connect to ORB [Root exception is org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No]
	at org.jboss.as.ejb3.deployment.processors.EjbLookupInjectionSource$1.getReference(EjbLookupInjectionSource.java:99)
	at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:143)
	... 221 more

      The EAP log of the node acting as client shows the following error (complete log in attached file node0.log): 

      2021-10-11 12:45:23,795 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-3) WFLYEJB0473: JNDI bindings for session bean named 'ClientEjb' in deployment unit 'deployment "client.jar"' are as follows:

	java:global/client/ClientEjb!org.jboss.as.test.iiop.basic.ClientEjb
	java:app/client/ClientEjb!org.jboss.as.test.iiop.basic.ClientEjb
	java:module/ClientEjb!org.jboss.as.test.iiop.basic.ClientEjb
	java:global/client/ClientEjb
	java:app/client/ClientEjb
	java:module/ClientEjb

2021-10-11 12:45:23,956 INFO  [org.jboss.weld.Version] (MSC service thread 1-4) WELD-000900: 3.1.6 (redhat)
2021-10-11 12:45:24,015 INFO  [org.jboss.as.arquillian] (MSC service thread 1-4) Arquillian deployment detected: ArquillianConfig[service=jboss.arquillian.config."client.jar",unit=client.jar,tests=[org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase]]
2021-10-11 12:45:24,763 INFO  [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0010: Deployed "client.jar" (runtime-name : "client.jar")
2021-10-11 12:45:25,406 WARNING [javax.enterprise.resource.corba._DEFAULT_.rpc.transport] (pool-12-thread-1) "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 54.236.40.220; port: 3628": org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No
	at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2200)
	at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2221)
	at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:223)
	at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:236)
	...
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:482)
	at sun.nio.ch.Net.connect(Net.java:474)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
	at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
	at org.wildfly.iiop.openjdk.security.SocketFactoryBase.createSocket(SocketFactoryBase.java:84)
	at org.wildfly.iiop.openjdk.security.NoSSLSocketFactory.createSocket(NoSSLSocketFactory.java:59)
	at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:207)
	... 240 more

      In the EAP log of the node acting as client we see (complete log in attached file node1.log): 

      2021-10-11 12:45:09,761 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-1) WFLYEJB0473: JNDI bindings for session bean named 'IIOPBasicBean' in deployment unit 'deployment "server.jar"' are as follows:

	java:global/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:app/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:module/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:jboss/exported/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	ejb:/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:global/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:app/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:module/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:jboss/exported/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:global/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicBean
	java:app/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicBean
	java:module/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicBean

      Please note that the IP logged in the error on node0:

      "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 54.236.40.220; port: 3628": org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No

      is actually the external IP of node1:

      • node0: private IP 172.31.4.194 - public IP 54.146.101.247 (acting as client)
      • node1: private IP 172.31.10.163 - public IP 54.236.40.220 (acting as server)

      Attachments

        1. as-iiop-testsuite-test-setup.sh
          8 kB
          Tommaso Borgato
        2. EC2_ONLY-node0-server.log
          707 kB
          Tommaso Borgato
        3. EC2_ONLY-node1-server.log
          144 kB
          Tommaso Borgato
        5. EC2_ONLY-test-node-ec2.log
          357 kB
          Tommaso Borgato
        6. node0.log
          1.97 MB
          Tommaso Borgato
        7. node0.stack.txt
          92 kB
          Tommaso Borgato
        8. node1.log
          178 kB
          Tommaso Borgato
        9. node1.stack.txt
          86 kB
          Tommaso Borgato
        10. org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase.txt
          6 kB
          Tommaso Borgato
        11. org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase-output.txt
          19 kB
          Tommaso Borgato
        12. test.log
          382 kB
          Tommaso Borgato
        13. test.stack.txt
          25 kB
          Tommaso Borgato
        14. TEST-org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase.xml
          100 kB
          Tommaso Borgato

        Activity

          People

            tadamski@redhat.com Tomasz Adamski
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: