Description of problem:

Validations within the component pipeline report issue with Horizon container/service and related endpoint.
Further investigation revealed that container ends in unhealthy state, with following syntax error as a culprit:

Trace:
------
2022-06-07T00:10:54.557914006+00:00 stderr F AH00526: Syntax error on line 25 of /etc/httpd/conf.d/10-horizon_vhost.conf:
2022-06-07T00:10:54.557954777+00:00 stderr F CustomLog takes two or three arguments, a file name, a custom log format string or format name, and an optional "env=" or "expr=" clause (see docs)

Nature of the file '10-horizon_vhost.conf' extracted from a machine running earlier release implies that problem may have appeared due to incorrectly escaped string. The relevant line contains several escaped quotes and if one of them were misplaced the function would receive different number of arguments.

Snip:

CustomLog "/var/log/httpd/horizon_access.log" "%a %l %u %t \"%r\" %>s %b \"%

i\" \"%

i\""

Complete file sample is available as attachment but should only serve as reference. The malformed file from failing container was not retrieved.

Version-Release number of selected component (if applicable):
N/A problem detected in the master branch CI.

How reproducible:
The issue appears to be constantly occurring for the past ~14 days.

Steps to Reproduce:
1. Use TripleO to deploy RHOS master(Zed) in OVB configuration, featureset 001, with 3 controller and one compute nodes as defined by periodic component pipeline job[0].
2. Observe status of Horizon container on any of the controller nodes. Eventually the Horizon container will fail with aforementioned syntax error.
The logs will be stored in file '/var/log/containers/stdouts/horizon.log.txt'.

Actual results:
Failed container with 'unhealthy' exit status.

Expected results:
Stable Horizon container that can be verified by validations.

Additional info:
Issue appears to be present in all jobs derived from the common ancestor[0] with exception of those using 'wallaby' or earlier branches.
At this time only VF specific component pipeline job[1] explicitly tests status of containers, and impact is therefore limited. However the issue is present in other jobs as well, only undetected.

[0]https://review.rdoproject.org/zuul/job/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master
[1]https://review.rdoproject.org/zuul/job/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation