Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2701

[Vector] [Cloudwatch] namespaceUUID is not added to logGroupName when forwarding logs to Cloudwatch.

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this fix, when forwarding logs to CloudWatch, namespaceUUID was not appended to the logGroupName. Now with the fix, namespaceUUID is appended, so a logGroupName in CloudWatch appears as
      "logGroupName": "vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286".
      Show
      Before this fix, when forwarding logs to CloudWatch, namespaceUUID was not appended to the logGroupName. Now with the fix, namespaceUUID is appended, so a logGroupName in CloudWatch appears as "logGroupName": "vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286".
    • Bug Fix
    • Log Collection - Sprint 235, Log Collection - Sprint 236, Log Collection - Sprint 239

      An issue for this is already opened in upstream by cahartma@redhat.com  https://github.com/vectordotdev/vector/issues/12824 and also mentioned in user story LOG-2223

      Created the issue to track the progress on bug fix. 

       

      Version of components:

      clusterlogging.v5.5.0

      elasticsearch-operator.v5.5.0

      Server Version: 4.10.0-0.nightly-2022-06-07-181847

      Kubernetes Version: v1.23.5+3afdacb

       

      Description of the issue:

      When forwarding logs to Cloudwatch using Vector as collector and specifying groupBy: namespaceUUID in the ClusterLogForwarder, the namespaceUUID is not added to the logGroupName in Cloudwatch.

       

      Steps to reproduce the issue:

      1 1 Create secret for forwarding to Cloudwatch.

      export ACCESS_KEY_ID=$(oc get secret aws-creds -n kube-system -o json | jq -r '.data.aws_access_key_id'|base64 -d)

export SECRET_ACCESS_KEY=$(oc get secret  aws-creds -n kube-system -o json |jq -r '.data.aws_secret_access_key'|base64 -d)

oc -n openshift-logging create secret generic cw-secret \
--from-literal=aws_access_key_id="${ACCESS_KEY_ID}" \
--from-literal=aws_secret_access_key="${SECRET_ACCESS_KEY}"

      2 Deploy a log generator app.

      oc new-project test

oc new-app https://raw.githubusercontent.com/openshift/verification-tests/master/testdata/logging/loggen/container_json_log_template.json

      3 Create ClusterLogForwarder instance to forward logs to CloudWatch.

      apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  outputs:
   - name: cw
     type: cloudwatch
     cloudwatch:
       groupBy: namespaceUUID
       groupPrefix: vectorcw
       region: us-east-2
     secret:
        name: cw-secret
  pipelines:
    - name: infra-logs
      inputRefs:
        - infrastructure
        - audit
        - application
      outputRefs:
        - cw

      4 Create ClusterLogging instance.

      apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance" 
  namespace: "openshift-logging"
spec:
  managementState: "Managed"  
  collection:
    logs:
      type: "vector"  
      vector: {}

      5 Check the log group in Cloudwatch. The app logs is missing namespaceUUID.

      aws logs describe-log-groups --log-group-name-prefix vectorcw --region=us-east-2

{
    "logGroups": [
        {
            "logGroupName": "vectorcw.",
            "creationTime": 1654668657739,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:301721915996:log-group:vectorcw.:*",
            "storedBytes": 0
        },
        {
            "logGroupName": "vectorcw.audit",
            "creationTime": 1654668566324,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:301721915996:log-group:vectorcw.audit:*",
            "storedBytes": 0
        },
        {
            "logGroupName": "vectorcw.infrastructure",
            "creationTime": 1654668566343,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:301721915996:log-group:vectorcw.infrastructure:*",
            "storedBytes": 0
        }
    ]
}

       

       

            [LOG-2701] [Vector] [Cloudwatch] namespaceUUID is not added to logGroupName when forwarding logs to Cloudwatch.

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Moderate: Logging Subsystem 5.7.4 - Red Hat OpenShift bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2023:4341

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Moderate: Logging Subsystem 5.7.4 - Red Hat OpenShift bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:4341

            Giriyamma Karagere Ramaswamy (Inactive) added a comment -

            Verified with logging 5.7.4, issue is fixed. The logGroupName in CloudWatch appears with namespaceUUID 

             

            $ oc get ns/test -oyaml | grep uid:
  uid: 15c2bc4d-0b05-4781-8ca2-6fb0154afe01
$ aws logs describe-log-groups --log-group-name-prefix vectorcw --region=us-east-2
{
    ...
         {
            "logGroupName": "vectorcw.15c2bc4d-0b05-4781-8ca2-6fb0154afe01",
            "creationTime": 1689855905008,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:301721915996:log-group:vectorcw.15c2bc4d-0b05-4781-8ca2-6fb0154afe01:*",
            "storedBytes": 0
        },
....
}

             

            Giriyamma Karagere Ramaswamy (Inactive) added a comment - Verified with logging 5.7.4, issue is fixed. The logGroupName in CloudWatch appears with namespaceUUID    $ oc get ns/test -oyaml | grep uid:   uid: 15c2bc4d-0b05-4781-8ca2-6fb0154afe01 $ aws logs describe-log-groups --log-group-name-prefix vectorcw --region=us-east-2 {     ...          {             "logGroupName" : "vectorcw.15c2bc4d-0b05-4781-8ca2-6fb0154afe01" ,             "creationTime" : 1689855905008,             "metricFilterCount" : 0,             "arn" : "arn:aws:logs:us-east-2:301721915996:log-group:vectorcw.15c2bc4d-0b05-4781-8ca2-6fb0154afe01:*" ,             "storedBytes" : 0         }, .... }  

            Jeffrey Cantrill added a comment -

            This issue requires Release Notes Text. Please modify the Release Note Text or set the Release Note Type to "None"

            Jeffrey Cantrill added a comment - This issue requires Release Notes Text. Please modify the Release Note Text or set the Release Note Type to "None"

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.7-rhel-8_upstream_7334fa5ea3ec1c1e993df2e989eebd52:

            Updated 2 upstream sources

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.7-rhel-8_ upstream _7334fa5ea3ec1c1e993df2e989eebd52 : Updated 2 upstream sources

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.7-rhel-8_upstream_c984f2336431fe7066011dc1a914bc18:

            Updated US source to: dbd1c60 Merge pull request #2057 from syedriko/syedriko-vector-0.28-release-5.7

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.7-rhel-8_ upstream _c984f2336431fe7066011dc1a914bc18 : Updated US source to: dbd1c60 Merge pull request #2057 from syedriko/syedriko-vector-0.28-release-5.7

            Jeffrey Cantrill added a comment -

            Set fix to 5.7 since we are backporting vector v0.28 to release-5.7. Placing back to POST for re-verification once all the backports needed merge

            Jeffrey Cantrill added a comment - Set fix to 5.7 since we are backporting vector v0.28 to release-5.7. Placing back to POST for re-verification once all the backports needed merge

            Giriyamma Karagere Ramaswamy (Inactive) added a comment -

            Verified with logging 5.8, issue is fixed.  The namespaceUUID is present in app logs.

            $ oc get ns/test -oyaml | grep uid:
  uid: f4af953a-e1be-46df-badc-d4ce9d289739

$ aws logs describe-log-groups --log-group-name-prefix vectorcw --region=us-east-2
{
    ...
        {
            "logGroupName": "vectorcw.f4af953a-e1be-46df-badc-d4ce9d289739",
            "creationTime": 1687253189150,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:301721915996:log-group:vectorcw.f4af953a-e1be-46df-badc-d4ce9d289739:*",
            "storedBytes": 0
        },
....
}

            Giriyamma Karagere Ramaswamy (Inactive) added a comment - Verified with logging 5.8, issue is fixed.  The namespaceUUID is present in app logs. $ oc get ns/test -oyaml | grep uid:   uid: f4af953a-e1be-46df-badc-d4ce9d289739 $ aws logs describe-log-groups --log-group-name-prefix vectorcw --region=us-east-2 {     ...         {             "logGroupName" : "vectorcw.f4af953a-e1be-46df-badc-d4ce9d289739" ,             "creationTime" : 1687253189150,             "metricFilterCount" : 0,             "arn" : "arn:aws:logs:us-east-2:301721915996:log-group:vectorcw.f4af953a-e1be-46df-badc-d4ce9d289739:*" ,             "storedBytes" : 0         }, .... }

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.8-rhel-9_upstream_0ac83fa7553f7989b59f010cb25a3607:

            Updated US source to: 14cd2a1 Merge pull request #1988 from syedriko/syedriko-syslog-vector-release-5.8

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.8-rhel-9_ upstream _0ac83fa7553f7989b59f010cb25a3607 : Updated US source to: 14cd2a1 Merge pull request #1988 from syedriko/syedriko-syslog-vector-release-5.8

            Ishwar Kanse added a comment -

            syedriko_sub@redhat.com Verified the fix with the images you provided and updated the QE test case https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-51978 , thanks. 

            Ishwar Kanse added a comment - syedriko_sub@redhat.com Verified the fix with the images you provided and updated the QE test case https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-51978 , thanks. 

            Sergey Yedrikov added a comment -

            Got the manual test above working:

            $ oc get ns/test -oyaml | grep uid:
  uid: b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286
$

$ aws logs describe-log-groups --log-group-name-prefix vectorcw --region=us-east-2
{
...
        {
            "logGroupName": "vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286",
            "creationTime": 1683084407196,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:269733383066:log-group:vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286:*",
            "storedBytes": 0
        },

...

            This is part of the upcoming update of vector and CLO to v0.28-1. If someone wants to give these a go, here are the images:

            • quay.io/syedriko/cluster-logging-operator:ns_uid
            • quay.io/syedriko/vector:ns_uid

            Sergey Yedrikov added a comment - Got the manual test above working: $ oc get ns/test -oyaml | grep uid: uid: b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286 $ $ aws logs describe-log-groups --log-group-name-prefix vectorcw --region=us-east-2 { ... { "logGroupName" : "vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286" , "creationTime" : 1683084407196, "metricFilterCount" : 0, "arn" : "arn:aws:logs:us-east-2:269733383066:log-group:vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286:*" , "storedBytes" : 0 }, ... This is part of the upcoming update of vector and CLO to v0.28-1. If someone wants to give these a go, here are the images: quay.io/syedriko/cluster-logging-operator:ns_uid quay.io/syedriko/vector:ns_uid

              syedriko_sub@redhat.com Sergey Yedrikov
              rhn-support-ikanse Ishwar Kanse
              Giriyamma Karagere Ramaswamy Giriyamma Karagere Ramaswamy (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: