In production, we ran the migrate_binding_scope migration with relation replication disabled as an optimization. Unfortunately, this will have resulted in relations becoming inconsistent with RBAC's database.

To handle system roles, that migration replicated two events for each group in any given tenant: the removal of all system roles from that group, then the readdition of all system roles to that group. (Due to the way the relevant code is implemented, this has the desired effect of putting all of the new role bindings in the correct scope for the system role's permissions.)

Later, this migration was re-run with replication disabled, as we believed that we only needed the database updates and not the Kessel updates. (I forget exactly why: to create RoleBinding models?)

When a role binding for a system role becomes unassigned (i.e. when its last group or user is removed), we delete the role binding (both locally and in Kessel). So, when we told the dual-write code that roles were being removed from a group, if that group happened to be the last group assigned a given system role in the tenant, the role binding for that role in that tenant would have been deleted. Then, when we told the dual-write code that the role was being re-added, it will have created a new role binding with a different ID. The relevant replication events being discarded will then have resulted in Kessel and RBAC having different role binding IDs.