Uploaded image for project: 'Data Foundation Bugs'
  1. Data Foundation Bugs
  2. DFBUGS-740

ceph-external-cluster using annotation to store encoded python script - issues being seen in both prometheus and loki as a result

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • odf-4.16.z
    • odf-4.16
    • ceph-monitoring
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ?
    • ?
    • None

      Redhat openshift data foundation documentation provides details of how to extract a python script from an annotation [1]

      That command being:

      ~~~
      oc get csv $(oc get csv -n openshift-storage | grep ocs-operator | awk '{print $1}') -n openshift-storage -o jsonpath='{.metadata.annotations.external\.features\.ocs\.openshift\.io/export-script}' | base64 --decode > ceph-external-cluster-details-exporter.py
      ~~~

      This annotation is a 110k encoded string, that is then visible in Prometheus targets output - causing that output to hang - we have a case for this
      We are also seeing this annotation being pulled into loglines being injested into Loki - 110k is large for a single logline [2]

      I think I understand that this script is being used to create serviceMonitors for external ceph clusters - so that, the external cluster metrics are present in the local prometheus...is there not another way this could be accomplished - without encoding a python script into an annotation?

      • which feels like borderline abuse of the suggested uses for an annotation [3] there is a paper written about how malicious code could be inserted into annotations [4]

      [1] - https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.9/html/upgrading_to_openshift_data_foundation/updating-the-openshift-data-foundation-external-secret_rhodf#updating-the-openshift-data-foundation-external-secret_rhodf

      [2] - https://redhat-internal.slack.com/archives/CJW8YBDR7/p1724093269102779
      [3] - https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
      [4] - https://www.lacework.com/blog/hiding-in-plaintext-sight-abusing-the-lack-of-kubernetes-auditing-policies
       

              aruniiird Arun Kumar Mohan
              rhn-support-nigsmith Nigel Smith
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated: