Post HBI migration in stage, we're seeing some inconsistencies with the number of hosts in HBI vs the number of host resources in Kessel

Before the migration, Tyler Creller noted there were 18 hosts in Kessel Inventory DB, most likely from testing our service

• On 11/17 i checked HBI's DB directly and immediately compared it to Kessel Inventory DB and there was a difference of 138 hosts where Kessel had more hosts than HBI

  insights=> select count(*) from hbi.hosts;
   count  
  --------
   195100
  (1 row)
  
  # VS
  
  $ gabi execute "select count(*) from reporter_resources where reporter_type = 'hbi' and  resource_type = 'host' and tombstone = 'false' and generation = '0'"
  [
    {
      "count": "195238"
    }
  ]
  

• On 11/18 using the same process, i am seeing a difference of 143 hosts where Kessel had more hosts than HBI, so the number is slowly climbing.

  insights=> select count(*) from hbi.hosts;
   count  
  --------
   195837
  (1 row)
  
  # VS
  
  $ gabi execute "select count(*) from reporter_resources where reporter_type = 'hbi' and  resource_type = 'host' and tombstone = 'false' and generation = '0'"
  [
    {
      "count": "195980"
    }
  ]
  

In reviewing the Kessel Inventory Consumer pod logs, there are no errors in message processing to indicate any failed calls to Inventory API to delete hosts. If there were, the consumer would also be stuck still trying to process those messages. This leads us to wonder if some deletes are not making it to the outbox by means of any kind of race condition or host deletion and outbox write not happening in same transaction/call.

On a side note, i have grabbed 400 random hosts over time in the last two days and compared them to entries in Inventory DB and all data has matched and looks good with no discrepencies!
Our main concern is the drift in hosts counts where Kessel Inventory is showing we have more hosts than HBI does