TEIID20020 Error establishing socket to host and port: <ip>:<port>. Reason: Read timed out

Hello, friends.

I've been struggling with the error below, since we wen to production a have increased the trafic on TEIID.

Our Wildfly based deployment has a "master" node responsible for Materializing(external) data from 25 remote sources (Multisource config) also using TEIID and using TEIID JDBC connection to them, and 2 worker nodes with external load balancing (HAProxy TCP mode) that provide data from the Materialized  views and direct remote access to others.

All 25 TEIID JDBC remote datasources use XA, along with external materialization in PostgresDB.

I'm able to connect to the web console anc check the current number of connections when the errors start(

How can we tackle this sort of error. Is this problem related to the number of threads or processors( 4 vCPUs machines for each node now) ?

(IJ031084: Unable to create connection): javax.resource.ResourceException: IJ031084: Unable to create connection
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:531)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory$1$1.run(XAManagedConnectionFactory.java:417)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory$1$1.run(XAManagedConnectionFactory.java:414)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory$1.run(XAManagedConnectionFactory.java:413)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory$1.run(XAManagedConnectionFactory.java:410)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:409)
        at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.open(XAResourceRecoveryImpl.java:355)
        at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.getXAResources(XAResourceRecoveryImpl.java:218)
        at com.arjuna.ats.internal.jbossatx.jta.XAResourceRecoveryHelperWrapper.getXAResources(XAResourceRecoveryHelperWrapper.java:51)
        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:627)
        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:235)
        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:182)
        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:770)
        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:382)
Caused by: org.teiid.jdbc.TeiidSQLException: TEIID20020 Error establishing socket to host and port: <server>:<port>. Reason: Read timed out
        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:131)
        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:67)
        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:62)
        at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:107)
        at org.teiid.jdbc.TeiidDataSource.getConnection(TeiidDataSource.java:272)
        at org.teiid.jdbc.TeiidDataSource.getXAConnection(TeiidDataSource.java:569)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:515)
        ... 16 more
Caused by: org.teiid.net.socket.SingleInstanceCommunicationException: TEIID20020 Error establishing socket to host and port: <server>:<port>. Reason: Read timed out
        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:134)
        at org.teiid.net.socket.SocketServerConnection.<init>(SocketServerConnection.java:88)
        at org.teiid.net.socket.SocketServerConnectionFactory.getConnection(SocketServerConnectionFactory.java:111)
        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:60)
        ... 20 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:92)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:493)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
        at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:113)
        at org.teiid.net.socket.SocketServerInstanceImpl.doHandshake(SocketServerInstanceImpl.java:124)
        at org.teiid.net.socket.SocketServerInstanceImpl.connect(SocketServerInstanceImpl.java:92)
        at org.teiid.net.socket.SocketServerConnectionFactory.getServerInstance(SocketServerConnectionFactory.java:91)
        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:122)
        ... 23 more