As an optimization, it makes sense to only create the AnnotatedType and assign it to ProcessAnnotatedType if the metadata has changed in anyway. Otherwise, you are wrapping the type just for the sake of wrapping it.
Therefore, I propose that any change to the state of the annotated type be tracked by a changed flag that is accessible by calling the AnnotatedTypeBuilder#isMetadataChanged() (or similarly named) method. It would be used as follows:
AnnotatedTypeBuilder b = AnnotatedTypeBuilder.readFromType(event.getAnnotatedType(), true);
// do extension logic