The Byteman agent requires certain permissions to execute; by default if you run within a security manager it will not get these permissions because the default policy does not allow it.
I would request that Byteman be enhanced to install a delegating Policy during its early bootstrap which grants Byteman itself AllPermission. I am not certain this will allow Byteman to work with -Djava.security.manager set, but for software which installs a SecurityManager programmatically (e.g. AS), it should work fine.
Here's an example which you could probably use, almost verbatim:
And the code which installs it: