When prepare on WS-AT participant timeouts (e.g. because of a communication error) such participant is marked as heuristic. That's not correct as timeouted communication during prepare is expected to be aborted.

This behaviour is confirmed by comments in the source code and tests expect such behaviour.

Unfortunately the returned error code makes opposite (marking participant as heuristic). The return code should do what the comment asks
https://github.com/jbosstm/narayana/blob/5.9.5.Final/XTS/WSCF/classes/com/arjuna/mwlabs/wscf/model/twophase/arjunacore/ParticipantRecord.java#L362

// if prepare timed out then we return error so it goes back on the
// prepare list and is rolled back

The same is correct for the cancel call on communication error. When cancel is called then it's expected to be retried and correct return code is needed to be returned.
See comment on the durable 2PC participant at https://github.com/jbosstm/narayana/blob/5.9.5.Final/XTS/WSTX/classes/com/arjuna/mwlabs/wst/at/participants/DurableTwoPhaseCommitParticipant.java#L232

// log an error here – if the participant is dead it will retry anyway