-
Bug
-
Resolution: Done
-
Normal
-
None
-
None
-
None
-
5
-
False
-
None
-
False
-
COST-3875 - Cloud: Show individual VM cost
-
-
In some cases, the EC2 report contains the same tag keys/values multiple times for the same instance.
"tags": [
{
"key": "tag1",
"values": [
"t1"
]
},
{
"key": "tag1",
"values": [
"t1"
]
},
{
"key": "tag2",
"values": [
"t2"
]
}
],
This seems to happen whenever some instance attribute changes over time - e.g., instance is tagged with a new tag, tag is removed or tag value changes throughout the month (the same holds for cost-categories - if they change, the tags are reported multiple times).
We need to make sure that each tag is reported just once per instance.
rh-ee-dnakabaa rhn-support-lcouzens
Additionally, some time ago we discussed the scenario of changing tag keys/values mid-month - based on my notes it looks like we wanted to show all tag keys/values that were used for a given instance in a given month - could you please confirm that this is still valid?
Steps to reproduce:
1. ingest AWS data with some tag/cost-category changed during the month
you can use the following yaml file:
---
generators:
- EC2Generator:
start_date: 2024-09-01
end_date: 2024-09-02
processor_arch: 64-bit
resource_id: "123"
product_sku: VEAJHRNKTEEE
region: eu-west-1b
operating_system: Ubuntu
tags:
resourceTags/user:tag1: t1
cost_category:
costCategory/category1: c1
costCategory/category2: c2
instance_type:
operating_system: Ubuntu
inst_type: m5.large
vcpu: "4"
physical_cores: 2.0
memory: '8 GiB'
storage: 'EBS Only'
family: 'General Purpose'
cost: 25.000
rate: 25.0
- EC2Generator:
start_date: 2024-09-02
processor_arch: 64-bit
resource_id: "123"
product_sku: VEAJHRNKTEEE
region: eu-west-1b
operating_system: Ubuntu
tags:
resourceTags/user:tag1: t1
resourceTags/user:tag2: t2
cost_category:
costCategory/category1: c1
costCategory/category2: c2
instance_type:
inst_type: m5.large
vcpu: "4"
physical_cores: 2.0
memory: '8 GiB'
storage: 'EBS Only'
family: 'General Purpose'
cost: 25.000
rate: 25.0
finalized_report:
invoice_id: 987654321
accounts:
payer: 9999999999999
user:
- 9999999999999
currency_code: NOK
2. check AWS EC2 report
http://localhost:8000/api/cost-management/v1/reports/aws/resources/ec2-compute/
you will see that tag1: t1 is duplicated
{
"resource_id": "i-123",
"date": "2024-09",
"usage": {
"value": 61.0,
"units": "Hrs"
},
"source_uuid": [
"090db3d7-54ea-48f9-b446-d1faf71afde2"
],
"account_alias": null,
"account": "9999999999999",
"instance_name": null,
"instance_type": "m5.large",
"operating_system": "Ubuntu",
"region": "eu-west-1",
"vcpu": 4,
"memory": "8 GiB",
"tags": [
{
"key": "tag1",
"values": [
"t1"
]
},
{
"key": "tag2",
"values": [
"t2"
]
},
{
"key": "tag1",
"values": [
"t1"
]
}
],
3. check content of reporting_awscostentrylineitem_summary_by_ec2_compute table in db
select * from reporting_awscostentrylineitem_summary_by_ec2_compute;
you will see two records differing just by tags (and obviously by usage/costs and dates)
As expected, both records contains "tag1": "t1" key-value pair, which is the reason for its duplication in EC2 report cause we probably pull all tags from the table at a moment.
-[ RECORD 1 ]--------------+---------------------------------------
uuid | 8cb141ef-86c0-423b-a897-b5cfdce33686
usage_start | 2024-09-02
usage_end | 2024-09-03
usage_account_id | 9999999999999
resource_id | i-123
instance_name |
instance_type | m5.large
operating_system | Ubuntu
region | eu-west-1
vcpu | 4
memory | 8 GiB
unit | Hrs
usage_amount | 37.000000000
normalization_factor | 0
normalized_usage_amount | 0
currency_code | NOK
unblended_rate | 25.000000000
unblended_cost | 925.000000000
markup_cost | 0.000000000
blended_rate | 25.000000000
blended_cost | 925.000000000
markup_cost_blended | 0.000000000000000
savingsplan_effective_cost | 0.000000000
markup_cost_savingsplan | 0.000000000000000
calculated_amortized_cost | 925.000000000
markup_cost_amortized | 0.000000000
public_on_demand_cost | 925.000000000
public_on_demand_rate | 25.000000000
tax_type |
tags | {"tag1": "t1", "tag2": "t2"}
source_uuid | 090db3d7-54ea-48f9-b446-d1faf71afde2
cost_category | {"category1": "c1", "category2": "c2"}
account_alias_id |
cost_entry_bill_id | 26
-[ RECORD 2 ]--------------+---------------------------------------
uuid | ac41ac4c-154b-406c-8f58-30ab6a56356b
usage_start | 2024-09-01
usage_end | 2024-09-01
usage_account_id | 9999999999999
resource_id | i-123
instance_name |
instance_type | m5.large
operating_system | Ubuntu
region | eu-west-1
vcpu | 4
memory | 8 GiB
unit | Hrs
usage_amount | 24.000000000
normalization_factor | 0
normalized_usage_amount | 0
currency_code | NOK
unblended_rate | 25.000000000
unblended_cost | 600.000000000
markup_cost | 0.000000000
blended_rate | 25.000000000
blended_cost | 600.000000000
markup_cost_blended | 0.000000000000000
savingsplan_effective_cost | 0.000000000
markup_cost_savingsplan | 0.000000000000000
calculated_amortized_cost | 600.000000000
markup_cost_amortized | 0.000000000
public_on_demand_cost | 600.000000000
public_on_demand_rate | 25.000000000
tax_type |
tags | {"tag1": "t1"}
source_uuid | 090db3d7-54ea-48f9-b446-d1faf71afde2
cost_category | {"category1": "c1", "category2": "c2"}
account_alias_id |
cost_entry_bill_id | 26
