This is hard to reproduce, it simply happens consistently on one CI server and doesn't happen on the different CI server. I haven't been unable to reproduce at all locally. I've been only able to debug it by introducing new debug logs and then running the full testsuite on the server where it fails.

The failing test cases are:

• org.jboss.as.test.integration.management.cli.InvalidAddressOperationTestCase.testValidationDisabled
• org.jboss.as.test.integration.management.cli.InvalidAddressOperationTestCase.testInvalidOperationWithParamValidationDisabled

The failure stacktrace as follows:

java.lang.IllegalArgumentException: Unexpected character 'F' while reading DMR stream
	at org.jboss.dmr.stream.ModelGrammarAnalyzer.newModelException(ModelGrammarAnalyzer.java:363)
	at org.jboss.dmr.stream.ModelReaderImpl.newModelException(ModelReaderImpl.java:861)
	at org.jboss.dmr.stream.ModelReaderImpl.next(ModelReaderImpl.java:615)
	at org.jboss.dmr.ModelNodeFactory.readFrom(ModelNodeFactory.java:86)
	at org.jboss.dmr.ModelNodeFactory.readFrom(ModelNodeFactory.java:67)
	at org.jboss.dmr.ModelNodeFactory.readFrom(ModelNodeFactory.java:72)
	at org.jboss.dmr.ModelNode.fromString(ModelNode.java:1740)
	at org.jboss.as.test.integration.management.cli.InvalidAddressOperationTestCase.assertFailedServerResponse(InvalidAddressOperationTestCase.java:131)
	at org.jboss.as.test.integration.management.cli.InvalidAddressOperationTestCase.assertFailedServerResponseAddress(InvalidAddressOperationTestCase.java:120)
	at org.jboss.as.test.integration.management.cli.InvalidAddressOperationTestCase.testValidationDisabled(InvalidAddressOperationTestCase.java:81)

The test case expects to catch an exception where the message is parseable Model response. Instead if gets an exception CommandFormatException where the message reads like:

Failed to get the list of the operation properties: "WFLYCTL0030: No resource definition is registered for address [
    ("subsystem" => "datasources"),
    ("bogus" => "*")
]"

The reason for a different exception appears to be that the test fails to disable the validate-operation-requests in the jboss-cli.xml, although it tries to do that by editing the EAP/bin/jboss-cli.xml file.

The hypothesis is that the jboss.cli.config system property is set to an unexpected value (appears to be set to empty string), which causes the CLI to not load the correct configuration file (which should be probably done from here: https://github.com/jbossas/wildfly-core-eap/blob/15.0.x/cli/src/main/java/org/jboss/as/cli/impl/CliConfigImpl.java#L106).

The jboss.cli.config system property is presumably set by a different test, so the failing/passing depends on the order of tests in the testsuite, which is undefined but tends to not change when the environment doesn't change (server, JDK, working directory, etc).

Clearing the jboss.cli.config system property appears to solve the problem on the problematic CI instance.