When I create a new workspace through HBI, and then I add some hosts to this workspace, I completely lose access to these hosts for up to 20 seconds (or less, this depends on the cache and staleness usage). What I think is happening is this: * I create a new group and HBI sends an API request to RBAC workspaces to also create the workspace there - there is no issue with sync between HBI and RBAC workspaces

• Since the response from RBAC is OK, HBI correctly creates the group in its own database and assigns the selected hosts to this new group. It also synchronizes the hosts association changes to Kessel.
• There is probably some delay (increased by cache + "flapping") until the new workspace gets synced from RBAC to Kessel (I'm not sure how RBAC - Kessel integration works, so please correct me if I'm wrong)
• When I make GET /hosts request on HBI, it queries Kessel to get "allowed workspaces" for Hosts View operation. Then, HBI returns only those hosts that have the association with one of the returned workspaces.
• Since the new workspace is not synced to Kessel yet, it was not returned by the ListAllowedWorkspaces method, and so HBI doesn't return this host. I'm not sure if fetching the host directly (which would use Check) would work, but my guess is that it would also fail, because the workspace isn't available in Kessel. Anyway, without the host being listed in the main page, a user wouldn't have a way of clicking on it.

This issue is reproducible in Stage via UI. To reproduce: * Go to HBI's main "Systems" page

• Select a couple of systems
• Click "Add to workspace"
• Click "Create workspace"
• Select a name and click "Create"
• Search for the new workspace and select it
• Click "Add"

The page will refresh and you will (most likely - depends on the timing) lose access to the selected hosts - you will not see them in the list. You can try refreshing the page until you see the hosts again.

The issue happens "only" up to 20 seconds after the workspace has been created, but this is a standard user workflow. It's not going to be hit only by automation, but by real Prod users that use UI as well. If you are not doing this the first time, then you can create the new workspace and add the hosts to it very quickly, and you will see this issue almost for sure. I refreshed the page 3 times before the hosts appeared again. I'm afraid this is going to be very annoying for users.
 
Original slack thread: https://redhat-internal.slack.com/archives/C08GTNAFZ55/p1773146453207389