-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.16
-
Moderate
-
None
-
False
-
Description of problem:
When ironic retrieves information from the `AttributeRegistry`, large debug logs can be generated. With enough nodes these can cause the log file to rotate almost immediately after ironic starts, resulting in the loss of valuable debugging information.
Steps to Reproduce:
Probably only reproducible on a real baremetal cluster
This was observed in https://issues.redhat.com/browse/OCPBUGS-50918 in the file inspect.local.6923788946813113617.tar.gz in ./namespaces/openshift-machine-api/pods/metal3-77bf6bdc56-hdwd7/metal3-ironic/metal3-ironic/logs/current.log
See lines similar to :
2025-02-07T11:39:48.032730997Z 2025-02-07 11:39:48.014 1 DEBUG sushy.resources.base [None req-adbddf09-8cee-42a6-8d16-b2d219a1a5a0 - - - - - -] Received representation of AttributeRegistry /redfish/v1/Registries/ManagerAttributeRegistry/ManagerAttributeRegistry.v1_0_0.json: {'_oem_vendors': None, 'description': 'This registry defines a representation of OEM Attribute instances', 'identi ....
this example was 2.5M long, but we don't have the ironic logs from then it was started 1 minute previous, presumably similar log entries caused the ironic logs to be rotated 1 minute after it was started
Actual results:
multiple long debug lines (23 bmh's in this case) cause ironic logs to rotate after restart
potential fixes could be to
- exclude these lines from being logged
- store them seperatly somewhere
- truncate the lines after a mac lenth is hit
- log them in a compressed and base64 encoded format (2.5M reduces to 79k in tests)
