Uploaded image for project: 'WildFly WIP'
  1. WildFly WIP
  2. WFWIP-437

Error running IIOP test module with remote EC2 nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Blocker
    • IIOP
    • None
    • Hide

      create two EC2 nodes with no networking restrictions e.g.

      aws ec2 run-instances --image-id ami-00753c9d2b9d4273c --count 2 --instance-type t3.medium --key-name jclouds-jenkins --security-groups allow-any

      note down for each instance:

      • Public IPv4 address: used later to configure the server
      • Private IPv4 address: used later to configure the test-suite

      configure the two nodes to run with the standalone-ec2-full-ha.xml profile (see Launching Clustered JBoss EAP), e.g.:

      ACCESS_KEY_ID='...'
SECRET_ACCESS_KEY='...'
S3_PING_BUCKET='...'
NODE_NAME='node0' # for the first node and 'node1' for the second node
EAP_PROFILE=standalone-ec2-full-ha.xml
JBOSS_HOME=/opt/rh/eap7/root/usr/share/wildfly
INTERNAL_IP_ADDRESS=`ip addr show | grep eth0 -A 2 | head -n 3 | tail -n 1 | awk '{ print $2 }' | sed "s-/24--g" | cut -d'/' -f1`
SERVICE_CONF_FILE=/etc/opt/rh/eap7/wildfly/eap7-standalone.conf

# Configure JBoss EAP to use the ec2-ha profile
cp /opt/rh/eap7/root/usr/share/wildfly/docs/examples/configs/$EAP_PROFILE $JBOSS_HOME/standalone/configuration/$EAP_PROFILE
chown jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE
chgrp jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE
chmod --reference=$JBOSS_HOME/standalone/configuration/standalone.xml $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# add managent user
./add-user.sh --user admin --password admin

# node name config
sudo sed -i 's/\(.*subsystem xmlns="urn:jboss:domain:undertow[^"]*"\)\(.*\)/\1 instance-id="${jboss.jvmRoute}"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# use tcp stack for clustering (using udp the cluster doens't get constituited)
sudo sed -i 's/\(.*channel name\="ee" stack\=\)"udp"\(.*\)/\1"tcp"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# https://access.redhat.com/solutions/2759881
sudo sed -i 's/\(.*\)<cluster password="[^"]*"\/>\(.*\)/\1<cluster password="admin" user="admin123"\/>\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE

# standalone-ec2-full-ha.xml
echo "WILDFLY_SERVER_CONFIG=$EAP_PROFILE" >> $SERVICE_CONF_FILE
echo "WILDFLY_BIND=$INTERNAL_IP_ADDRESS" >> $SERVICE_CONF_FILE

# node0
echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf

# node1 needs to have a port-offset=100 according to arquillian.xml file
echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.socket.binding.port-offset=100 -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf

echo "" >> $JBOSS_HOME/bin/standalone.conf

systemctl enable eap7-standalone
systemctl start eap7-standalone

      configure the AS test-suite to work with running EAP instances;

      e.g. in jboss-eap-7.4-src/testsuite/integration/iiop/src/test/config/arq/arquillian.xml add the following on every configured node:

      <property name="allowConnectingToRunningServer">true</property>
<property name="username">admin</property>
<property name="password">admin</property>

      Also the following property needs to be adjusted in order to pint to the JBoss EAP installation shipped with the AMI

      <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=${basedir}/target/jbossas-iiop-client</property>

      <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=/opt/rh/eap7/root/usr/share/wildfly</property>

      build the project

      e.g.

      ./build.sh clean install -DskipTests -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository -Denforcer.skip=true
./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true

      run the IIOP test module (replace the IPs with the actual IPs of your EC2 nodes)

      e.g.

      ./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true -Dnode0=3.85.211.95 -Dnode1=3.223.141.15 -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository
      Show
      create two EC2 nodes with no networking restrictions e.g. aws ec2 run-instances --image-id ami-00753c9d2b9d4273c --count 2 --instance-type t3.medium --key-name jclouds-jenkins --security-groups allow-any note down for each instance: Public IPv4 address: used later to configure the server Private IPv4 address: used later to configure the test-suite configure the two nodes to run with the standalone-ec2-full-ha.xml profile (see Launching Clustered JBoss EAP ), e.g.: ACCESS_KEY_ID='...' SECRET_ACCESS_KEY='...' S3_PING_BUCKET='...' NODE_NAME='node0' # for the first node and 'node1' for the second node EAP_PROFILE=standalone-ec2-full-ha.xml JBOSS_HOME=/opt/rh/eap7/root/usr/share/wildfly INTERNAL_IP_ADDRESS=`ip addr show | grep eth0 -A 2 | head -n 3 | tail -n 1 | awk '{ print $2 }' | sed "s-/24--g" | cut -d'/' -f1` SERVICE_CONF_FILE=/etc/opt/rh/eap7/wildfly/eap7-standalone.conf # Configure JBoss EAP to use the ec2-ha profile cp /opt/rh/eap7/root/usr/share/wildfly/docs/examples/configs/$EAP_PROFILE $JBOSS_HOME/standalone/configuration/$EAP_PROFILE chown jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE chgrp jboss $JBOSS_HOME/standalone/configuration/$EAP_PROFILE chmod --reference=$JBOSS_HOME/standalone/configuration/standalone.xml $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # add managent user ./add-user.sh --user admin --password admin # node name config sudo sed -i 's/\(.*subsystem xmlns="urn:jboss:domain:undertow[^"]*"\)\(.*\)/\1 instance-id="${jboss.jvmRoute}"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # use tcp stack for clustering (using udp the cluster doens't get constituited) sudo sed -i 's/\(.*channel name\="ee" stack\=\)"udp"\(.*\)/\1"tcp"\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # https://access.redhat.com/solutions/2759881 sudo sed -i 's/\(.*\)<cluster password="[^"]*"\/>\(.*\)/\1<cluster password="admin" user="admin123"\/>\2/' $JBOSS_HOME/standalone/configuration/$EAP_PROFILE # standalone-ec2-full-ha.xml echo "WILDFLY_SERVER_CONFIG=$EAP_PROFILE" >> $SERVICE_CONF_FILE echo "WILDFLY_BIND=$INTERNAL_IP_ADDRESS" >> $SERVICE_CONF_FILE # node0 echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf # node1 needs to have a port-offset=100 according to arquillian.xml file echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.socket.binding.port-offset=100 -Djboss.jgroups.s3_ping.access_key='$ACCESS_KEY_ID' -Djboss.jgroups.s3_ping.secret_access_key='$SECRET_ACCESS_KEY' -Djboss.jgroups.s3_ping.bucket='$S3_PING_BUCKET' -Djboss.jvmRoute=$NODE_NAME -Djboss.bind.address=$INTERNAL_IP_ADDRESS -Djboss.bind.address.private=$INTERNAL_IP_ADDRESS -Djboss.bind.address.management=$INTERNAL_IP_ADDRESS\"" >> $JBOSS_HOME/bin/standalone.conf echo "" >> $JBOSS_HOME/bin/standalone.conf systemctl enable eap7-standalone systemctl start eap7-standalone configure the AS test-suite to work with running EAP instances; e.g. in jboss-eap-7.4-src/testsuite/integration/iiop/src/test/config/arq/arquillian.xml add the following on every configured node: <property name="allowConnectingToRunningServer">true</property> <property name="username">admin</property> <property name="password">admin</property> Also the following property needs to be adjusted in order to pint to the JBoss EAP installation shipped with the AMI <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=${basedir}/target/jbossas-iiop-client</property> <property name="javaVmArguments">${server.jvm.args} -Djboss.inst=/opt/rh/eap7/root/usr/share/wildfly</property> build the project e.g. ./build.sh clean install -DskipTests -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository -Denforcer.skip=true ./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true run the IIOP test module (replace the IPs with the actual IPs of your EC2 nodes) e.g. ./integration-tests.sh clean install -Dts.iiop -Dts.noSmoke -Denforcer.skip=true -Dnode0=3.85.211.95 -Dnode1=3.223.141.15 -Dmaven.repo.local=jboss-eap-7.4.1.GA-maven-repository/maven-repository

    Description

      We run the IIOP test module in the following scenario:

      • we created two EC2 nodes using AMI with name RHEL-7-JBEAP-7.4.0_HVM_GA-20210909-x86_64-0-Access2-GP2
      • on a node external to AWS (e.g. a Fedora laptop) we run the IIOP AS test-suite module

      The test log shows the following error (complete log in attached file test.log): 

      [INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase
[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 17.489 s <<< FAILURE! - in org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase
[ERROR] testHomeHandle(org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase)  Time elapsed: 1.437 s  <<< ERROR!
javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:268)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:390)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:160)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	...
Caused by: javax.naming.NamingException: WFLYNAM0062: Failed to lookup env/home [Root exception is java.lang.RuntimeException: javax.naming.NamingException: WFLYIIOP0031: Cannot connect to ORB [Root exception is org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No]]
	at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:159)
	at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:83)
	at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:207)
	at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:193)
	at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:189)
	at org.jboss.as.naming.deployment.ContextNames$BindInfo$1$1.getReference(ContextNames.java:322)
	... 216 more
Caused by: java.lang.RuntimeException: javax.naming.NamingException: WFLYIIOP0031: Cannot connect to ORB [Root exception is org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No]
	at org.jboss.as.ejb3.deployment.processors.EjbLookupInjectionSource$1.getReference(EjbLookupInjectionSource.java:99)
	at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:143)
	... 221 more

      The EAP log of the node acting as client shows the following error (complete log in attached file node0.log): 

      2021-10-11 12:45:23,795 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-3) WFLYEJB0473: JNDI bindings for session bean named 'ClientEjb' in deployment unit 'deployment "client.jar"' are as follows:

	java:global/client/ClientEjb!org.jboss.as.test.iiop.basic.ClientEjb
	java:app/client/ClientEjb!org.jboss.as.test.iiop.basic.ClientEjb
	java:module/ClientEjb!org.jboss.as.test.iiop.basic.ClientEjb
	java:global/client/ClientEjb
	java:app/client/ClientEjb
	java:module/ClientEjb

2021-10-11 12:45:23,956 INFO  [org.jboss.weld.Version] (MSC service thread 1-4) WELD-000900: 3.1.6 (redhat)
2021-10-11 12:45:24,015 INFO  [org.jboss.as.arquillian] (MSC service thread 1-4) Arquillian deployment detected: ArquillianConfig[service=jboss.arquillian.config."client.jar",unit=client.jar,tests=[org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase]]
2021-10-11 12:45:24,763 INFO  [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0010: Deployed "client.jar" (runtime-name : "client.jar")
2021-10-11 12:45:25,406 WARNING [javax.enterprise.resource.corba._DEFAULT_.rpc.transport] (pool-12-thread-1) "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 54.236.40.220; port: 3628": org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No
	at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2200)
	at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2221)
	at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:223)
	at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:236)
	...
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:482)
	at sun.nio.ch.Net.connect(Net.java:474)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
	at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
	at org.wildfly.iiop.openjdk.security.SocketFactoryBase.createSocket(SocketFactoryBase.java:84)
	at org.wildfly.iiop.openjdk.security.NoSSLSocketFactory.createSocket(NoSSLSocketFactory.java:59)
	at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:207)
	... 240 more

      In the EAP log of the node acting as client we see (complete log in attached file node1.log): 

      2021-10-11 12:45:09,761 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-1) WFLYEJB0473: JNDI bindings for session bean named 'IIOPBasicBean' in deployment unit 'deployment "server.jar"' are as follows:

	java:global/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:app/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:module/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:jboss/exported/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	ejb:/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicRemote
	java:global/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:app/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:module/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:jboss/exported/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicHome
	java:global/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicBean
	java:app/server/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicBean
	java:module/IIOPBasicBean!org.jboss.as.test.iiop.basic.IIOPBasicBean

      Please note that the IP logged in the error on node0:

      "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 54.236.40.220; port: 3628": org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No

      is actually the external IP of node1:

      • node0: private IP 172.31.4.194 - public IP 54.146.101.247 (acting as client)
      • node1: private IP 172.31.10.163 - public IP 54.236.40.220 (acting as server)

      Attachments

        1. TEST-org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase.xml
          100 kB
        2. test.stack.txt
          25 kB
        3. test.log
          382 kB
        4. org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase-output.txt
          19 kB
        5. org.jboss.as.test.iiop.basic.BasicIIOPInvocationTestCase.txt
          6 kB
        6. node1.stack.txt
          86 kB
        7. node1.log
          178 kB
        8. node0.stack.txt
          92 kB
        9. node0.log
          1.97 MB
        10. EC2_ONLY-test-node-ec2.log
          357 kB
        11. EC2_ONLY-surefire-reports.zip
          99 kB
        12. EC2_ONLY-node1-server.log
          144 kB
        13. EC2_ONLY-node0-server.log
          707 kB
        14. as-iiop-testsuite-test-setup.sh
          8 kB

        Activity

          People

            tadamski@redhat.com Tomasz Adamski
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: