Diagram:

Changes:

1. Add new In Progress and Sent statuses in the remittance status (these status are new and will have no effects in the existing logic for now)
2. Add a new configuration property to query remittances that are stuck. Default value is 1 day.
   1. When a remittance is stuck?
      1. when status is In Progress and updated_at is older than 1 day
      2. when status is Sent and updated_at is older than 1 day
3. Add two new counters: one to count the number of entities from sent to unknown and another one to count the number of entities from in progress to failed.

Logic:

Stream all the remittances in "In Progress" or "Sent" status which "updated_at" is higher than the configurable property (one day by default)

For each remittance:

• if the status is "In Progress", we update it to "Failed" and error_code="error_sending_to_aggregate_topic", increment the counter and add a INFO trace for better troubleshooting.
  •  We set to "failed" status because this remittance never made it to the producers. Setting the value to 'failed' allows the next remittance to include the failed value and prevent under billing.
• if the status is "Sent", we update it to "Unknown", increment the counter and add a INFO trace for better troubleshooting.

Tests:

• Create the new API and write component tests to verify the above logic

Acceptance Criteria

• Update the openapi.yaml with remittance reconciliation. This is an internal and test related api roles/permissions.
• Configuration property that reconciles the remittances to one day.
• New counters in grafana dashboard.
• Calling the new api transition to new statuses for stuck remittance records. 
• New error_code is searchable in splunk through logs