CloudWatch fluentd plugin posts all logs it forwards to AWS to its own log

• While running the PerfScale fluentd to CloudWatch log test (https://docs.google.com/document/d/10vv_SVC7fUammkvdn6-05bGrzdAtXxiJ_Z6QMzfUL4A/edit#heading=h.h8lgt35i9r4a) I noticed that all the messages that are being generated by the log generator pod are being replicated in the fluentd container logs.

 

This would cause an increase in memory usage (which we already are hitting the limit on; see https://issues.redhat.com/browse/LOG-1059) and CPU (which we hit at 100% during testing when we unbound the fluentd pod memory usage).

 

A snip from the current fluentd pod logs:

```

\"kubernetes\":

{\"container_name\":\"log-generator2\",\"namespace_name\":\"log-generator\",\"pod_name\":\"log-generator-pbhsq\",\"container_image\":\"quay.io/dry923/log_generator:latest\",\"container_image_id\":\"quay.io/dry923/log_generator@sha256:143fd046876838d1f8e4acff828f9f71e7da7472df348f407e92e561014dec6d\",\"pod_id\":\"b56cfa57-28f5-41be-a30b-ec09b6a6a40a\",\"host\":\"ip-10-0-224-163.us-west-2.compute.internal\",\"master_url\":\"https://kubernetes.default.svc\",\"namespace_id\":\"5134c2df-a339-4eae-be8e-75753329102f\",\"flat_labels\":[\"controller-uid=98ef04c1-df8e-4aca-9f94-66bf8b54ae59\",\"job-name=log-generator\",\"name=log-generator\"]}

,\"message\":\"NWTJEKR56QIAQ5VINCQP0WQNLTJH5IX2XGBHSB8DM8C16OQA3B3VRN8PXMN783MVYKVZN43KT0SGP0BXTSRADE4192ACMSPBLOKNXG22MMEE28WN84O60ETQ7FD158IU9FCP7P0KLC83S0I6JPPI9K68EDSL99M975W81HP16BVKMX5T8WCYEGSF6AFO41JHVQR74KK9KA1570MXWYBF6IJ5W12VIPXWDJMWY6EIVJTPOD4XAY76HDI95QEU3A9FEN3GOWTODZH3ZA27XY4ZLT8DG1Q4" ... (2181 bytes)>}],log_group_name:"dry-logtest-7mbsj.application",log_stream_name:"kubernetes.var.log.containers.log-generator-pbhsq_log-generator_log-generator2-9cb0c46bf5e2860fa3ad7a39c2376bcaf878e672191328e1f9828e5cdd1e8341.log",sequence_token:"49609932663458872515816355989557366195202703253053985314")

```