Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-15200

[RHOSO 18] Podman Logs Not Written to Configured File Path (`/var/log/containers/stdouts/nova_compute.log`)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Undefined Undefined
    • None
    • None
    • openstack-nova
    • 0
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Low

      To Reproduce 

      1. Prepare an RHOSO 18.0 test environment, deployment refers to the doc https://ci-framework.pages.redhat.com/docs/main/ci-framework/deploy_va.html
      $ sudo virsh list --all
       Id   Name                       State
      ------------------------------------------
       8    cifmw-ocp-master-0         running
       9    cifmw-ocp-master-1         running
       10   cifmw-ocp-master-2         running
       11   cifmw-compute-72gp44i9-0   running
       12   cifmw-compute-72gp44i9-1   running
       13   cifmw-compute-72gp44i9-2   running
       14   cifmw-controller-0         running 
      1. Inspect the container log configuration on compute-1
      $ sudo podman inspect nova_compute | grep -i log
                          "Log": null
                     "CAP_SYSLOG",
                          "Source": "/var/log/containers/nova",
                          "Destination": "/var/log/containers/nova",
                          "config_data": "{'image': 'quay.io/podified-antelope-centos9/openstack-nova-compute@sha256:797ae8df69745e3294b884f79b59d02c7c55869e5088033aa021225e0178f13c', 'privileged': True, 'user': 'nova', 'restart': 'always', 'command': 'kolla_start', 'net': 'host', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/var/lib/openstack/config/nova:/var/lib/kolla/config_files:ro', '/var/lib/openstack/cacerts/nova/tls-ca-bundle.pem:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,z', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/dev:/dev', '/var/log/containers/nova:/var/log/containers/nova', '/var/lib/libvirt:/var/lib/libvirt', '/run/libvirt:/run/libvirt:shared', '/var/lib/nova:/var/lib/nova:shared', '/var/lib/iscsi:/var/lib/iscsi:z', '/etc/multipath:/etc/multipath:z', '/etc/multipath.conf:/etc/multipath.conf:ro', '/etc/iscsi:/etc/iscsi:ro', '/etc/nvme:/etc/nvme', '/var/lib/openstack/config/ceph:/var/lib/kolla/config_files/ceph:ro', '/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro']}",
                          "config_data={'image': 'quay.io/podified-antelope-centos9/openstack-nova-compute@sha256:797ae8df69745e3294b884f79b59d02c7c55869e5088033aa021225e0178f13c', 'privileged': True, 'user': 'nova', 'restart': 'always', 'command': 'kolla_start', 'net': 'host', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/var/lib/openstack/config/nova:/var/lib/kolla/config_files:ro', '/var/lib/openstack/cacerts/nova/tls-ca-bundle.pem:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,z', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/dev:/dev', '/var/log/containers/nova:/var/log/containers/nova', '/var/lib/libvirt:/var/lib/libvirt', '/run/libvirt:/run/libvirt:shared', '/var/lib/nova:/var/lib/nova:shared', '/var/lib/iscsi:/var/lib/iscsi:z', '/etc/multipath:/etc/multipath:z', '/etc/multipath.conf:/etc/multipath.conf:ro', '/etc/iscsi:/etc/iscsi:ro', '/etc/nvme:/etc/nvme', '/var/lib/openstack/config/ceph:/var/lib/kolla/config_files/ceph:ro', '/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro']}",
                          "--log-driver",
                          "--log-level",
                          "--log-opt",
                          "path=/var/log/containers/stdouts/nova_compute.log",
                          "/var/log/containers/nova:/var/log/containers/nova",
                          "/var/log/containers/nova:/var/log/containers/nova:rw,rprivate,rbind",
                     "LogConfig": {
                          "Path": "/var/log/containers/stdouts/nova_compute.log",
       

      We can see the output
      "Path": "/var/log/containers/stdouts/nova_compute.log",

      1. Check the log file 
      $ ls -lh /var/log/containers/stdouts/nova_compute.log
      ls: cannot access '/var/log/containers/stdouts/nova_compute.log': No such file or directory
      
      $ sudo podman exec -it nova_compute ls -l /var/log/containers/stdouts/nova_compute.log
      ls: cannot access '/var/log/containers/stdouts/nova_compute.log': No such file or directory
       

      The file is either empty or does not exist.

      1. Check logs using podman and systemd
      sudo podman logs nova_compute
      sudo journalctl -u edpm_nova_compute -xe 

      Logs are available via these commands, but are not written in the configured file.

      Actual behavior

      Logs are only accessible via `podman logs` and `journalctl`, but `/var/log/containers/stdouts/nova_compute.log` is not being updated.  

      Expected behavior

      • Update the configuration of "Path" since it doesn't take effect. 

      Screenshots

      • none

      Device Info (please complete the following information):

      • none

      Bug impact

      • none

      Known workaround

      • none

      Additional context

      • Check the log driver
      $ sudo podman inspect nova_compute | grep -i "log-driver"
                          "--log-driver",
       
      • If the log driver is `journald`, this may indicate that the log path configuration is ignored. 

              Unassigned Unassigned
              juzhou@redhat.com Junqin Zhou
              rhos-workloads-compute
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: