-
Bug
-
Resolution: Done-Errata
-
Major
-
Logging 5.8.0
-
False
-
None
-
False
-
NEW
-
NEW
-
Prior to this update, the logging must-gather can not gather any logs on the FIPS enabled cluster. A new oc client is now available in cluster-logging-rhel9-operator and must-gather works as expected on FIPS clusters
-
Bug Fix
-
Proposed
-
-
-
Log Collection - Sprint 242, Log Collection - Sprint 244, Log Collection - Sprint 245, Log Collection - Sprint 246, Log Collection - Sprint 247, Log Collection - Sprint 248, Log Collection - Sprint 249
-
Important
Description of problem:
CLO must-gather can not gather any data. Debug logs as below cat must-gather.local.4013599789171133369/registry-redhat-io-openshift-logging-cluster-logging-rhel9-operator-sha256-9395cd98d934be428132fb3da33f61f5c2e746027152b8cf34dc7c552345c5e4/gather-debug.log 2023-08-02 02:19:55 BEGIN inspecting CRs... 2023-08-02 02:19:55 BEGIN inspecting CR ns/openshift-operator-lifecycle-manager ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting CR ns/openshift-operator-lifecycle-manager ... 2023-08-02 02:19:55 BEGIN inspecting CR ns/openshift-logging ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting CR ns/openshift-logging ... 2023-08-02 02:19:55 BEGIN inspecting CR ns/openshift-operators-redhat ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting CR ns/openshift-operators-redhat ... 2023-08-02 02:19:55 BEGIN inspecting CR nodes ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting CR nodes ... 2023-08-02 02:19:55 BEGIN inspecting CR clusterroles ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting CR clusterroles ... 2023-08-02 02:19:55 BEGIN inspecting CR clusterrolebindings ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting CR clusterrolebindings ... 2023-08-02 02:19:55 BEGIN inspecting CR persistentvolumes ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting CR persistentvolumes ... 2023-08-02 02:19:55 END inspecting CRs... 2023-08-02 02:19:55 BEGIN inspecting namespaces ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-logging/pods ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-logging/pods ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-logging/roles ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-logging/roles ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-logging/rolebindings ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-logging/rolebindings ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-logging/configmaps ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-logging/configmaps ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-logging/serviceaccounts ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-logging/serviceaccounts ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-logging/events ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-logging/events ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operator-lifecycle-manager/pods ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operator-lifecycle-manager/pods ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operator-lifecycle-manager/roles ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operator-lifecycle-manager/roles ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operator-lifecycle-manager/rolebindings ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operator-lifecycle-manager/rolebindings ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operator-lifecycle-manager/configmaps ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operator-lifecycle-manager/configmaps ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operator-lifecycle-manager/serviceaccounts ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operator-lifecycle-manager/serviceaccounts ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operator-lifecycle-manager/events ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operator-lifecycle-manager/events ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operators-redhat/pods ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operators-redhat/pods ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operators-redhat/roles ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operators-redhat/roles ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operators-redhat/rolebindings ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operators-redhat/rolebindings ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operators-redhat/configmaps ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operators-redhat/configmaps ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operators-redhat/serviceaccounts ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operators-redhat/serviceaccounts ... 2023-08-02 02:19:55 BEGIN inspecting namespace openshift-operators-redhat/events ... FIPS mode is enabled, but the required OpenSSL library is not available 2023-08-02 02:19:55 END inspecting namespace openshift-operators-redhat/events ... 2023-08-02 02:19:55 END inspecting namespaces ... 2023-08-02 02:19:55 BEGIN inspecting install resources ... 2023-08-02 02:19:55 Skipping install inspection. No CLO or EO deployment found 2023-08-02 02:19:55 END inspecting install resources ... 2023-08-02 02:19:55 Skipping collection inspection. No CLO found 2023-08-02 02:19:55 Skipping logstorage inspection. No Elasticsearch deployment found
Version-Release number of selected component (if applicable):
cluster-logging-rhel9-operator/images/v5.8.0-101
How reproducible:
Always
Steps to Reproduce:
- Deploy Logging 5.8 and clusterlogging/instance
- oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')
- is related to
-
-
- Closed
-
- links to
-
RHBA-2024:128809
Logging Subsystem 5.9.0 - Red Hat OpenShift
- mentioned on
[LOG-4403] CLO must-gather doesn't work on FIPS enabled cluster
Anping Li
added a comment - Verified using cluster-logging-rhel9-operator/images/v5.9.0-59 
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.9-rhel-9_upstream_80d8650ce9aa2671f7a4d4d299282f19:
Updated US source to: a9916c6
LOG-4884: explicitly set env variable KUBERNETES_SERVICE_HOST=kubernetes.default.svc
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.9-rhel-9_ upstream _80d8650ce9aa2671f7a4d4d299282f19 : Updated US source to: a9916c6 LOG-4884 : explicitly set env variable KUBERNETES_SERVICE_HOST=kubernetes.default.svc Updated Dockerfile in upstream: https://github.com/openshift/cluster-logging-operator/pull/2319
Updated Dockerfile in midstream: https://gitlab.cee.redhat.com/openshift-logging/log-collection-midstream/-/blob/openshift-logging-5.9-rhel-9/distgit/containers/cluster-logging-operator/Dockerfile.in?ref_type=heads
Solution proposed by Arda Guclu
registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cli-artifacts:v4.16.0-202312130634.p0.g36f5a45.assembly.stream
There is an `oc.rhel9` located at `/usr/share/openshift/linux_amd64`
We need an oc client, built on RHEL9 so that we can run oc commands on a FIPS cluster. There are several issues out there, but so far the fix seems to be using oc adm extract with a new --command=oc.rhel9 to extract the updated binary. But, we can't run oc, so we can't extract
https://issues.redhat.com/browse/OCPBUGS-23386
https://issues.redhat.com/browse/OCPBUGS-25461
We need an image that replaces this in our dockerfile (containing the rebuilt oc tool).
FROM registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cli:v4.16.0-202312130634.p0.g36f5a45.assembly.stream AS origincli
The latest cli image is still using openssl linking from rhel8. image=registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cli-artifacts:v4.16.0-202401041032.p0.gdff7dc3.assembly.stream
# rpm -qf /usr/lib64/libcrypto.so.1.1 openssl-libs-1.1.1k-9.el8_6.x86_64 # rpm -qf /lib64/libcrypto.so.3 error: file /lib64/libcrypto.so.3: No such file or directory
Based on the jiras linked above, this should be resolved in the next few days.
REMINDER: we need to change this in BOTH upstream AND midstream
When ART creates a release, some artifacts are extracted from payload images. At the moment, these are all compiled on a rhel8 environment.
Being resolved in:
https://issues.redhat.com//browse/OCPBUGS-23386
https://issues.redhat.com/browse/OCPBUGS-25461
GitLab CEE Bot
added a comment - Sergey Yedrikov mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch syedriko-log-4403-3-release-5.8:
[release-5.8]
LOG-4403: CLO must-gather doesn't work on FIPS enabled cluster, oc from builder
GitLab CEE Bot
added a comment - Sergey Yedrikov mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch syedriko-log-4403-3-release-5.8 : [release-5.8] LOG-4403 : CLO must-gather doesn't work on FIPS enabled cluster, oc from builder GitLab CEE Bot
added a comment - Sergey Yedrikov mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch syedriko-log-4403-2-release-5.8:
[release-5.8]
LOG-4403: CLO must-gather doesn't work on FIPS enabled cluster, openshift-clients
GitLab CEE Bot
added a comment - Sergey Yedrikov mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch syedriko-log-4403-2-release-5.8 : [release-5.8] LOG-4403 : CLO must-gather doesn't work on FIPS enabled cluster, openshift-clients 
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Logging for Red Hat OpenShift - 5.9.0), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2024:1591