-
Feature
-
Resolution: Done
-
Normal
-
Logging 5.6
-
False
-
False
-
SRES-Arch
-
Undefined
Goals
- Provide a new output option to forward logs to Splunk.
Non-Goals
Motivation
Usually, we recommend to use Splunk Connect for Kubernetes but some customers have requirements to send logs to multiple, different systems including Splunk. For these use cases, they'd like to avoid deploying multiple different "Agents" and want to use our supported solution instead.
Alternatives
For forwarding logs only to Splunk, users should use Splunk Connect for Kubernetes.
Acceptance Criteria
Risk and Assumptions
Documentation Considerations
Open Questions
Additional Notes
Original Request =========================================
Proposed title of this feature request
Support Splunk as output for ClusterLogForwarder
What is the nature and description of the request?
Splunk is one of the big players in logging management and nowadays there's no direct support to it in ClusterLogForwarder.
Why does the customer need this?
Customer has a mix of external Elasticsearch and Splunk clusters and would like to send the logs directly from Fluentd to it using HEC via the fluent-plugin-splunk-hec plugin, which is already present in the redhat fluentd image used on OCP4.
They're aware of valid solutions like setting an external Fluentd and forward the logs from there to Splunk, but considering how popular is Splunk makes sense to make it a supported ouptut.
List any affected packages or components.
Cluster Logging
[OBSDA-85] Support Splunk as output for ClusterLogForwarder
The feature is being assessed in the next team planning meeting. Updates to follow.
Albert Wyatt
added a comment - There is an issue with only supporting syslog, and that is that customers using Splunk-Cloud would need to to deploy a syslog server with a Splunk Heavy Forwarder to encrypt the logs as they are sent to Splunk-Cloud. This isn't really an option for OpenShift Dedicated customers that may only consume OpenShift as a cloud service and not also have additional virtual machines in a VPC that they do not manage.
Fluentd already supports a HEC token: https://github.com/splunk/fluent-plugin-splunk-hec and shipping directly to Splunk as an output.
Albert Wyatt
added a comment - There is an issue with only supporting syslog, and that is that customers using Splunk-Cloud would need to to deploy a syslog server with a Splunk Heavy Forwarder to encrypt the logs as they are sent to Splunk-Cloud. This isn't really an option for OpenShift Dedicated customers that may only consume OpenShift as a cloud service and not also have additional virtual machines in a VPC that they do not manage.
Fluentd already supports a HEC token: https://github.com/splunk/fluent-plugin-splunk-hec and shipping directly to Splunk as an output. 
Sergio Garcia Martinez
added a comment - Yeah, I didn't realized about the syslog option you mentioned, I apologize. At this point I'm not sure about the situation of the original customer that raised this RFE (I'll check with the engineer which is handling the case) but if your decission is firm, then go ahead.
Sergio Garcia Martinez
added a comment - Yeah, I didn't realized about the syslog option you mentioned, I apologize. At this point I'm not sure about the situation of the original customer that raised this RFE (I'll check with the engineer which is handling the case) but if your decission is firm, then go ahead. sgarciam@redhat.com I believe you have missed my question. Splunk is capable of accepting logs via the syslog protocol and doesn't require any other resources; they can use our fluentd. We made the decision, intentionally, to support Splunk as a target via the syslog protocol as a development and support optimization. Please consider this usecase when responding to my original question
Sergio Garcia Martinez
added a comment - jcantril@redhat.com they're able to do it, but it's a waste of resources as it requires to deploy new agents on each node whereas we already have a fluentd which can collect the logs and use the plugin (which is already included in the image) to do the job.
On the other hand, it might be difficult to explain why some 3rd parties targets are supported and other (commonly accepted and widely known in the log industry as Splunk) are not.
Sergio Garcia Martinez
added a comment - jcantril@redhat.com they're able to do it, but it's a waste of resources as it requires to deploy new agents on each node whereas we already have a fluentd which can collect the logs and use the plugin (which is already included in the image) to do the job.
On the other hand, it might be difficult to explain why some 3rd parties targets are supported and other (commonly accepted and widely known in the log industry as Splunk) are not. sgarciam@redhat.com Before we make an acceptance determination please explain why the customer is not able to use syslog to forward to Splunk. There was an intentional decision with our previous PM to explicitly to only support Splunk over syslog so that we didnt have to support an additional protocol.
Hi Gerd,
I have added this issue to the Logging 5.5 release grooming. Once I have had a chance to review with engineering I will provided an update.
Shannon
Ather Adil
added a comment - rhn-support-edrich - please expedite this one. I see follow up comments from account teams but missing the traction from engineering org
Ather Adil
added a comment - rhn-support-edrich - please expedite this one. I see follow up comments from account teams but missing the traction from engineering org 
w00t w00t! thank you very much!