Loading...

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: Logging 5.5.0
Affects Version/s: Logging 5.4.0
Component/s: Log Collection
Labels:
- NAPS
- devel_ack+

Blocked:
False
Blocked Reason:
None
Ready:
False
Epic Link:
Enable Vector to use OpenSSL
Docs QE Status:
NEW
Feature Link:
OBSDA-111 - Provide GA Support for Vector Collector with OpenShift Logging
Flagged:

Impediment
QE Status:
VERIFIED

Sprint:
Logging (Core) - Sprint 218, Logging (Core) - Sprint 219, Logging (Core) - Sprint 220, Log Collection - Sprint 221

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Version of components:

Cluster Logging 5.4

Server Version: 4.9.0-0.nightly-2022-04-07-205533

Kubernetes Version: v1.22.5+a36406b

Description of the problem:

On a FIPS enabled cluster Vector collector pods fail to start with the below error:

Apr 08 05:06:05.059 ERROR vector::topology: Configuration error. error=Sink "default": Could not build PKCS#12 archive for identity: error:0607B0C8:digital envelope routines:EVP_CipherInit_ex:disabled for FIPS:crypto/evp/evp_enc.c:227:, error:06074078:digital envelope routines:EVP_PBE_CipherInit:keygen failure:crypto/evp/evp_pbe.c:131:, error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:crypto/pkcs12/p12_decr.c:41:, error:2306C067:PKCS12 routines:PKCS12_item_i2d_encrypt:encrypt error:crypto/pkcs12/p12_decr.c:144:, error:23073067:PKCS12 routines:PKCS12_pack_p7encdata:encrypt error:crypto/pkcs12/p12_add.c:119:

Steps to reproduce the issue:

1 Deploy a FIPS enabled OCP cluster.

2 Install the 5.4 CLusterLogging and Elasticsearch operators.

3 Create a ClusterLogging instance.

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance" 
  namespace: "openshift-logging"
  annotations:
    logging.openshift.io/preview-vector-collector: enabled
spec:
  managementState: "Managed"  
  logStore:
    type: "elasticsearch"  
    retentionPolicy: 
      application:
        maxAge: 10h
      infra:
        maxAge: 10h
      audit:
        maxAge: 10h
    elasticsearch:
      nodeCount: 3 
      storage: {} 
      resources: 
          limits:
            memory: "4Gi"
          requests:
            memory: "1Gi"
      proxy: 
        resources:
          limits:
            memory: 256Mi
          requests:
            memory: 256Mi
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"  
    kibana:
      replicas: 1
  collection:
    logs:
      type: "vector"  
      vector: {}

4 Check the collector pod logs are in CrashLoopBackOff with the following error in logs.

$ oc get pods
NAME                                               READY   STATUS             RESTARTS         AGE
cluster-logging-operator-6c884f55f6-2t2xc          1/1     Running            0                31m
collector-68jdr                                    1/2     CrashLoopBackOff   10 (4m33s ago)   30m
collector-cpb8q                                    1/2     CrashLoopBackOff   10 (4m35s ago)   30m
collector-kmzvb                                    1/2     CrashLoopBackOff   10 (4m10s ago)   30m
collector-nkfp8                                    1/2     CrashLoopBackOff   10 (4m38s ago)   30m
collector-nlc9p                                    1/2     CrashLoopBackOff   10 (4m15s ago)   30m
collector-sw4zx                                    1/2     CrashLoopBackOff   10 (4m36s ago)   30m
collector-wbxxl                                    1/2     CrashLoopBackOff   10 (4m26s ago)   30m
elasticsearch-cdm-gd7smnhe-1-574db95c8d-r86k5      2/2     Running            0                30m
elasticsearch-cdm-gd7smnhe-2-6bbf478ccd-hmnzx      2/2     Running            0                30m
elasticsearch-cdm-gd7smnhe-3-754d56c65b-n49v8      2/2     Running            0                30m
elasticsearch-im-app-27489930--1-wl65c             0/1     Completed          0                12m
elasticsearch-im-audit-27489930--1-vtx9h           0/1     Completed          0                12m
elasticsearch-im-infra-27489930--1-r2d6s           0/1     Completed          0                12m
kibana-797c59589c-kfvlg                            2/2     Running            0                30m
loki-operator-controller-manager-5789bb7b8-lt6vd   2/2     Running            0                31m

$ oc logs collector-nlc9p -c collector 
Apr 08 05:37:57.381  INFO vector::app: Log level is enabled. level="info"
Apr 08 05:37:57.381  INFO vector::app: Loading configs. path=[("/etc/vector/vector.toml", Some(Toml))]
Apr 08 05:37:57.384  INFO vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="ip-10-0-178-157.us-east-2.compute.internal"
Apr 08 05:37:57.393 ERROR vector::topology: Configuration error. error=Sink "default": Could not build PKCS#12 archive for identity: error:0607B0C8:digital envelope routines:EVP_CipherInit_ex:disabled for FIPS:crypto/evp/evp_enc.c:227:, error:06074078:digital envelope routines:EVP_PBE_CipherInit:keygen failure:crypto/evp/evp_pbe.c:131:, error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:crypto/pkcs12/p12_decr.c:41:, error:2306C067:PKCS12 routines:PKCS12_item_i2d_encrypt:encrypt error:crypto/pkcs12/p12_decr.c:144:, error:23073067:PKCS12 routines:PKCS12_pack_p7encdata:encrypt error:crypto/pkcs12/p12_add.c:119: