Ideally the test would be something like.

Set up a configuration of a web app secured using the caching realm which in turn wraps something like the filesystem realm with some identities defined.

Verify a successful authentication.
Change the credential for the identity.
Verify the original credential still works.
Clear the cache
Now verify the new identity.

The test could also consider attributes which are based on the underlying realm, i.e. do we see the cached values or the most recent values.