While deploying an update to liquibase to add a PK UUID to billable_usage_remittance, there were code failures where billable_usage_remittances failed to insert because they did not have the UUID from the old deployment. When this happened the kafka billable usage message still sent even though the database transaction failed. We need to investigate why this is happening and have the messages send only when the db records are inserted successfully.

https://docs.spring.io/spring-kafka/reference/kafka/transactions.html#transaction-synchronization

Splunk query to see the column contraint violation at 27 March:

index=rh_rhsm namespace=rhsm-prod "ERROR: null value in column "uuid" violates not-null constraint"