Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3610

Audit logs are not forwarded when logging is deployed on ROSA hypershift

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Log Collection - Sprint 231

      Description of problem:

      Audit logs are missing on CW/default ES end. 

      Version-Release number of selected component (if applicable):

      ROSA hypershift/RedHat Hosted cluster

      rosa version: 1.2.11

      How reproducible: 

      Always

      Steps to Reproduce:

      1. Create ROSA hypershift cluster on us-west-2 region
      2. Deploy 5.6.0 CLO and ES operatorS
      3. Set up CLF to forward logs to default ES/Cloudwatch
      4. Create ClusterLogging instance with either vector or fluentd as collector

      Actual results: 

      Infra and application logs found on CW/Default ES.

      Audit logs are missing

      Expected results:

      Infra, Audit and application logs found on CW/Default ES.

      Additional info:

      CLF: http://pastebin.test.redhat.com/1091100

      fluent.conf: http://pastebin.test.redhat.com/1091101

      vector.toml: http://pastebin.test.redhat.com/1091102

      $ rosa version
      1.2.11
      There is a newer release version, please consider updating: https://github.com/openshift/rosa/releases/tag/v1.2.13$ rosa list cluster
      ID                                NAME            STATE
      21kebqrhpssasq3qaiahtrjfp7qvv882  kbharti-loghs1  ready
      $ rosa describe cluster -c 21kebqrhpssasq3qaiahtrjfp7qvv882     
      Name:                       kbharti-loghs1
      ID:                         21kebqrhpssasq3qaiahtrjfp7qvv882
      External ID:                1180f035-822b-4d22-9070-f39742db0f36
      Control Plane:              Red Hat hosted
      OpenShift Version:          4.12.0
      Channel Group:              stable
      DNS:                        kbharti-loghs1.xo93.s3.devshift.org
      AWS Account:                301721915996
      API URL:                    https://api.hcp.kbharti-loghs1.xo93.s3.devshift.org:443Console URL:                https://console-openshift-console.apps.kbharti-loghs1.rosa.kbharti-loghs1.xo93.s3.devshift.orgRegion:                     us-west-2
      Multi-AZ:                   falseNodes:
       - Compute (workers):           6
      Network:
       - Type:                    OVNKubernetes
       - Service CIDR:            172.30.0.0/16
       - Machine CIDR:            10.0.0.0/16
       - Pod CIDR:                10.128.0.0/14
       - Host Prefix:             /23
      STS Role ARN:               arn:aws:iam::301721915996:role/ManagedOpenShift-Installer-Role
      Support Role ARN:           arn:aws:iam::301721915996:role/ManagedOpenShift-Support-Role
      Instance IAM Roles:
       - Control plane:           arn:aws:iam::301721915996:role/ManagedOpenShift-ControlPlane-Role
       - Worker:                  arn:aws:iam::301721915996:role/ManagedOpenShift-Worker-Role
      Operator IAM Roles:
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-openshift-ingress-operator-cloud-credentials
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-kube-system-kube-controller-manager
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-kube-system-capa-controller-manager
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-openshift-image-registry-installer-cloud-cre
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-openshift-cloud-credential-operator-cloud-cr
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-openshift-cloud-network-config-controller-cl
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-kube-system-control-plane-operator - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-kube-system-kms-provider
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-openshift-cluster-csi-drivers-ebs-cloud-cred
       - arn:aws:iam::301721915996:role/kbharti-loghs1-j9o8-openshift-machine-api-aws-cloud-credentials
      State:                      ready 
      Private:                    No
      Created:                    Feb  2 2023 03:54:23 UTC
      Details Page:               https://qaprodauth.console.redhat.com/openshift/details/s/2LANjV6sYh8Wzy57baD3PibTng3OIDC Endpoint URL:          https://d3gt1gce2zmg3d.cloudfront.net/21kebqrhpssasq3qaiahtrjfp7qvv882$ oc get csv -n openshift-logging                                                                                             
      NAME                            DISPLAY                            VERSION   REPLACES   PHASE
      cluster-logging.v5.6.0          Red Hat OpenShift Logging          5.6.0                Succeeded
      elasticsearch-operator.v5.6.0   OpenShift Elasticsearch Operator   5.6.0                Succeeded
      kbharti@kbharti-mac ~ % oc get pods -n openshift-logging
      NAME                                            READY   STATUS      RESTARTS   AGE
      cluster-logging-operator-6cd5987b95-9n6s7       1/1     Running     0          5h49m
      collector-4wf7v                                 2/2     Running     0          15s
      collector-9dkfq                                 2/2     Running     0          15s
      collector-bz6hn                                 2/2     Running     0          15s
      collector-cvsrq                                 2/2     Running     0          15s
      collector-km58g                                 2/2     Running     0          15s
      collector-vlg2p                                 2/2     Running     0          15s
      elasticsearch-cdm-nox9qnx3-1-5c545879d6-gbrg5   2/2     Running     0          5h45m
      elasticsearch-im-app-27922200-fvdvj             0/1     Completed   0          9m14s
      elasticsearch-im-audit-27922200-sbmms           0/1     Completed   0          9m14s
      elasticsearch-im-infra-27922200-ft6cf           0/1     Completed   0          9m14s
      kibana-5bfd68b475-w9kz9                         2/2     Running     0          5h45m
      CW logstreams:
      $ ~ % aws --output json logs describe-log-groups | jq 
      {
        "logGroups": [
         
      {       "logGroupName": "kbharti-rosahs-log.application",       "creationTime": 1675311905699,       "metricFilterCount": 0,       "arn": "arn:aws:logs:us-west-2:301721915996:log-group:kbharti-rosahs-log.application:*",       "storedBytes": 0     }
      ,
         
      {       "logGroupName": "kbharti-rosahs-log.infrastructure",       "creationTime": 1675311839160,       "metricFilterCount": 0,       "arn": "arn:aws:logs:us-west-2:301721915996:log-group:kbharti-rosahs-log.infrastructure:*",       "storedBytes": 0     }
        ]
      }
      ES indices
      sh-4.4$ indices
      Thu Feb  2 10:11:21 UTC 2023
      health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
      green  open   app-000005   0gOIpxKQRDO7F9nG7VvGaQ   1   0      18002            0          5              5
      green  open   .security    GK2OoahnS_OWaiiQ_iHZuw   1   0          6            0          0              0
      green  open   .kibana_1    GCf8U3tFSwG8lRtWy6ITGQ   1   0          0            0          0              0
      green  open   app-000004   YqkvYx8sRjKPeGCQBtas1A   1   0      14360            0          4              4
      green  open   app-000006   xiljwSqQS5-6MV98ov2Q1A   1   0       6302            0          1              1
      green  open   app-000002   jODEi6CGQaCivMVO2KFOzQ   1   0      14388            0          4              4
      green  open   audit-000001 8D2jIqyaTDuq7cc_tYWToA   1   0          0            0          0              0
      green  open   app-000003   MJn8m6vBT6Webs1unr6ZQQ   1   0      14382            0          4              4
      green  open   infra-000001 w8IyQEPzRj6KSk6UC3mamA   1   0     435117            0        292            292
      green  open   app-000001   9bbJTCOiSSmHuiZhiF9Stg   1   0      16516            0          4              4
      

              cahartma@redhat.com Casey Hartman
              rhn-support-kbharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: