Problem statement:

Quay.io ships logs to their AWS accounts using syslog-cloudwatch-bridge. This tooling is out of date and will probably raise some compliance red flags in the future so we should switch Quay to use a more standardized solution, or the Cluster Logging Operator.

Resources:

Slack discussion started here: https://redhat-internal.slack.com/archives/C057QJFQJA1/p1733491329249779

Cluster Logging Operator (CLO) is installed by AppSRE on each cluster onboarded to A-I and normally sends cluster logs to our app-sre-logs AWS account.

Quay openshift-logging namespaces:

• https://gitlab.cee.redhat.com/service/app-interface/-/blob/aac446ded4065b594d3ac1405dd6d38391d57f34/data/openshift/quayp04ue2/namespaces/openshift-logging.yaml
• https://gitlab.cee.redhat.com/service/app-interface/-/blob/aac446ded4065b594d3ac1405dd6d38391d57f34/data/openshift/quayp05ue1/namespaces/openshift-logging.yaml
• https://gitlab.cee.redhat.com/service/app-interface/-/blob/aac446ded4065b594d3ac1405dd6d38391d57f34/data/openshift/quays02ue1/namespaces/openshift-logging.yaml

Implementation ideas:

This ticket will consist of several different components and will require collaboration with Quay team:

• CLO makes use of templates for deployment so you will need to make modifications to the templates, allowing the IAM service account for shipping logs to be provisioned on another account besides app-sre-logs
• You may need to update other resources provisioned in the openshift-logging namespace to account for these changes

Acceptance Criteria:

• Work with the Quay team to update their application templates to remove the syslog-cloudwatch image and disable shipping of logs via the bridge
• Update quayp04ue2, quayp05ue1, quays02ue1 CLO configuration to ship application logs to quayio-prod/quayio-stage AWS accounts and validate the configuration
• Archive our fork of the syslog-cloudwatch-bridge repo
  • Remove any other jobs/references in App-Interface as well

Default Acceptance Criteria:

• All existing/affected SOPs have been updated
• New SOPs have been written
• The feature has both unit and end to end tests passing in all test pipelines and through upgrades
• If the feature requires QE involvement, QE has signed off
• The feature exposes metrics necessary to manage it (VALET/RED)
• The feature has had a security review
• Contract impact assessment
• Documentation is complete