Description of problem:

When defined in an output `spec.outputs.<output>.tls.insecureSkipVerify: true`  with http instead of https as:

$ oc get clusterlogforwarder instance -o yaml -n openshift-logging
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  creationTimestamp: "2023-12-14T08:29:56Z"
  generation: 3
  name: instance
  namespace: openshift-logging
  resourceVersion: "1003988"
  uid: 71bcc54e-5a9e-4f60-a1b0-c5db3fd180cc
spec:
  outputs:
  - name: splunk
    tls:
      insecureSkipVerify: true
    type: splunk
    url: http://splunk:8088
  pipelines:
  - inputRefs:
    - application
    name: container-logs
    outputRefs:
    - splunk

Then, the collectors pods dissappear:

$ oc get pods -l component=collector -n openshift-logging
No resources found in openshift-logging namespace.

And not error is visible in the `clusterLogForwarder` indicating any error in the pipeline:

$ oc get clusterlogforwarder instance -o yaml -n openshift-logging
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  creationTimestamp: "2023-12-14T08:29:56Z"
  generation: 3
  name: instance
  namespace: openshift-logging
  resourceVersion: "1003988"
  uid: 71bcc54e-5a9e-4f60-a1b0-c5db3fd180cc
spec:
  outputs:
  - name: splunk
    tls:
      insecureSkipVerify: true
    type: splunk
    url: http://splunk:8088
  pipelines:
  - inputRefs:
    - application
    name: container-logs
    outputRefs:
    - splunk
status:
  conditions:
  - lastTransitionTime: "2023-12-14T08:32:42Z"
    status: "True"
    type: Ready

Version-Release number of selected component (if applicable):

$ oc get csv -n openshift-logging|grep -i logging
cluster-logging.v5.8.1                  Red Hat OpenShift Logging          5.8.1     cluster-logging.v5.8.0                  Succeeded 

How reproducible:

Always

Steps to Reproduce:

1. Create a clusterlogforwarder instance where it's used `insecureSkipVerify: true` and the url of the output to `http` instead of `https` as:

   kind: ClusterLogForwarder
   metadata:
     name: instance 
     namespace: openshift-logging 
   spec:
     outputs:
     - name: splunk
       tls:
         insecureSkipVerify: true
       type: splunk
       url: http://splunk:8088
     pipelines: 
      - name: container-logs
        inputRefs:
        - application
        outputRefs:
        - splunk
   

Actual results:

1.  Review the `clusterLogForwarder` status where no error visible, but also, if it's created like the example, not status section:

   $ oc get clusterlogforwarder instance -o yaml -n openshift-logging
   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     creationTimestamp: "2023-12-14T08:29:56Z"
     generation: 7
     name: instance
     namespace: openshift-logging
     resourceVersion: "1021542"
     uid: 71bcc54e-5a9e-4f60-a1b0-c5db3fd180cc
   spec:
     outputs:
     - name: splunk
       tls:
         insecureSkipVerify: true
       type: splunk
       url: http://splunk:8088
     pipelines:
     - inputRefs:
       - application
       - infrastructure
       - audit
       name: container-logs
       outputRefs:
       - splunk
   

1. Not error present in the `clusterlogging` operator pod pointing to a wrong configuration

   $ oc logs $(oc get pod -l name=cluster-logging-operator -n openshift-logging -o name ) -n openshift-logging
   

Expected results:

Able to see two possible options:

1. If it's http instead of https, then, it's generated a valid configuration for the collector using the not secure endpoint http and omitting the `spec.outputs.<output>.tls.insecureSkipVerify: true`
2. Don't allow to write a `clusterlogforwarder` configuration where it's together http and `spec.outputs.<output>.tls.insecureSkipVerify: true`, and/or throwing an error in the status section indicating that the options used are invalid together

Probably, the best option should be the second for a better User Experience perspective because it highlight that an invalid configuration is present and make it to be reviewed again and decide if removing `spec.outputs.<output>.tls.insecureSkipVerify: true`, or use `https`.

Workaround

`insecureSkipVerify: true` is not an option to be used with http:

spec:
  outputs:
  - name: splunk
    tls:
      insecureSkipVerify: true
    type: splunk
    url: http://splunk:8088

If the server defined is really listening in a not secure way, this is `http`, then removing `spec.outputs.<output>.tls.insecureSkipVerify: true`:

spec:
  outputs:
  - name: splunk
    type: splunk
    url: http://splunk:8088

Or in case that the output is really listening in a secure port, then, change `http` by `https`:

spec:
  outputs:
  - name: splunk
    tls:
      insecureSkipVerify: true
    type: splunk
    url: https://splunk:8088

NOTE: it's not recommended to use `spec.outputs.<output>.tls.insecureSkipVerify: true`, then, as the CA is always public, it could be get as follow:

// server goes without https in front
$ server=<server>
$ port=<port>
$ echo | openssl s-client -connect ${server}:${port} -servername ${server} | openssl x509 -out ca.pem

And create a secret containing the CA for being used by the collector to stablish the network connection. For how to configure this:
https://docs.openshift.com/container-platform/4.14/logging/log_collection_forwarding/log-forwarding.html#cluster-logging-collector-log-forwarding-about_log-forwarding