Uploaded image for project: 'Hybrid Cloud Console'
  1. Hybrid Cloud Console
  2. RHCLOUD-45441

Investigate serialization errors in inventory-api outbox

XMLWordPrintable

    • Product / Portfolio Work
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • Unset
    • None

      It has become normal to see a significant number of outbox-related DB serialization errors in the inventory-api logs. For example:

      2026/02/26 13:39:30 /workspace/internal/data/outboxeventsrepository.go:16 ERROR: could not serialize access due to read/write dependencies among transactions (SQLSTATE 40001) 

      These errors seem to relate to the delete in the code:

      func PublishOutboxEvent(tx *gorm.DB, event *model_legacy.OutboxEvent) error {
      	if err := tx.Create(event).Error; err != nil {
      		return err
      	}
      	if err := tx.Delete(event).Error; err != nil {
      		return err
      	}
      	return nil
      } 

      It is understood that this error can occur from time to time due to the use of Postgres' Serializable isolation level; however, it seems to be happening a lot. After a recent inventory-api deployment, it was observed in the logs of 1 pod 223 times in 7 minutes.

      This task is to investigate why this is happening and to do something about it.

      Slack thread:

      https://redhat-internal.slack.com/archives/C059NUD49PD/p1772113343941489?thread_ts=1772034632.534889&cid=C059NUD49PD 

              anatale.openshift Antony Natale
              mmclaugh@redhat.com Mark McLaughlin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: