-
Spike
-
Resolution: Done
-
Normal
-
None
-
Quality / Stability / Reliability
-
False
-
-
False
-
None
-
Unset
-
None
-
-
Analyse:
1) Between the 6th and the 7th Notifications received 23405 events from Polices for the org "5894300". 11 polices are configured on this org, each message was linked to at least one of those polices.2) The 7th at midnight, Notifications engine pod aggregated those 23405 events according the Policies rules and generated a db record of 161 MB with following structure:
event_data
└── application_aggregated_data_list[] (array) [1 element]
├── app_name
└── aggregated_data
├── end_time
├── start_time
├── unique_system_count
└── policies (object) [11 UUID keys]
└── {policy_uuid} (e.g., "1fd7d293-0047-4cb6-b88f-4b457ce44d9b")
├── policy_id
├── policy_name
├── policy_description
├── policy_condition
├── unique_system_count
└── hosts[] (array) [varies: 14,050 to 23,405 per policy, 248,079 total]
├── display_name
├── inventory_id
└── tags[] (array) [~10 tags per host]
├── key
└── value
3) notifications-connector-email pod was unable to load and transform those data into an email with available Memory, It generated a kind of poison pile situation on this kafka partition.
4) when the available memory was increased, the notifications-connector-email pod was able to load and process this record.
