Scenario:
1) Enlist XAResource
2) Network failure occurs (example from tests: a proxy is killed)
3) Prepare is called (or commit in case of 1PC) -> RMERR is thrown, the table gets locked
4) Proxy is started
5) Attempt to read the modified table waits infinitely 

From what I understand, the fact that the table gets locked is correct. I compared it with MsSQL and the scenario is very similar BUT, the table unlocks when the transaction timeout runs out.

This timeout is set in wildfly-transaction-client/LocalTransaction.java#L176 by method XAResource.setTransactionTimeout.

The problem seems to lie in the Sybase driver's implementation of setTransationTimeout as it has no effect apparently. As you can see in the setTransactionTimeout description, the success of this function is determined by the returned value and Sybase driver always return false.

I don't think that we can get that driver "fixed" any time soon, or whether there is something else we can do to assure that the table gets unlocked, but we should have this at least documented.
Something like https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.4/html-single/configuration_guide/index#vendor_specific_xa_recovery comes to mind.

There is a driver property DEFAULT_QUERY_TIMEOUT, that can be set to avoid the infinite waiting to read the blocked table, but the table will still stay locked.
I also found some mentions of transaction timeout in Sybase ASE 12.5.1 EJB Server User’s Guide, but it seems that this is configurable on DB server side and I don't know anything about DB server configuration.