Right now, in entitlements POST /seats a client can only add one user to a seat. This means they have to call this endpoint multiple times for multiple users. If they do this asynchronously, they run into risk of hitting a race condition with AMS. An example of this would be: 2 requests, a and b, fire at once. The quota version will be retrieved for request a, then request b, then request a will update the quota version, then request b will fail to update the quota version because its current version is out of date. So right now clients have to make these multiple requests synchronously. Slack thread for an example of this happening: https://redhat-internal.slack.com/archives/C04MZT67Y9M/p1696267201240929?thread_ts=1696256860.529229&cid=C04MZT67Y9M

To fully avoid hitting this race condition, we would have to either implement retry logic in entitlements to check for the race condition, or AMS would have to provide db locking or something around the quota version (probably way more complicated so ignore this option).

I have never implemented retry logic in go so this ticket is to investigate what it could look like, and implement it if we like the solution. So in entitlements we would wrap retry logic around the call to GetQuotaCost and QuotaAuthorization, so if we see that the quota version is outdated in the QuotaAuthorization call, wait some amount of time then call GetQuotaCost again to get the new version and try again. Do this n # of times before failing and giving up.

Rough ideas of settings would be try 3 times, and wait 1s, then 2s, then 3s between each try. Make these settings configurable