Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-9878

neutron/openvswitch directories of logs missing from edpm nodes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • rhos-18.0.1
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ?
    • ?
    • Yes
    • Important

      rh-ee-fyanac brought to my attention there are recent failures started on neutron component job [1].

      Failure seems like regression.

      Traceback suggests file either doesn't exist or empty, since number should be extracted from log entry as string, and typecasted to int.

      After debugging on similar job held, it seems on edpm nodes where logs are stored, only global system messages log file '/var/log/messages' used [2] but of not other log files that were used in the past such as [3].

      After checking it more with egarciar@redhat.com,
      it seems neutron/openvswitch directories of logs are not present on edpm nodes under '/var/log/containers/'.

      [1]

      https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/main/component-network-edpm-rhel9-rhoso18.0-crc/39ff4f6/controller/ci-framework-data/tests/test_operator/tempest-tests-single-thread-testing-workflow-step-1/stestr_results.html

      Failing:

      test_dropped_and_accepted_traffic_logged
      test_only_accepted_traffic_logged

      Traceback (most recent call last):
        File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/whitebox_neutron_tempest_plugin/tests/scenario/test_security_group_logging.py", line 902, in test_dropped_and_accepted_traffic_logged
          self._test_dropped_and_accepted_traffic_logged()
        File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/whitebox_neutron_tempest_plugin/tests/scenario/test_security_group_logging.py", line 761, in _test_dropped_and_accepted_traffic_logged
          self.start_track_log(vm_a['hv_ssh_client'])
        File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/whitebox_neutron_tempest_plugin/tests/scenario/test_security_group_logging.py", line 253, in start_track_log
          self._get_logs_and_counts(hypervisor_ssh, start_track=True)
        File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/whitebox_neutron_tempest_plugin/tests/scenario/test_security_group_logging.py", line 211, in _get_logs_and_counts
          _track_value = int(hypervisor_ssh.exec_command(
      ValueError: invalid literal for int() with base 10: ''

      [2] 

      [zuul@compute-0 ~]$ sudo grep -IErin acl_log /var/log/
      /var/log/messages:118809:Sep  3 07:52:03 np0001998469 ovn_controller[66565]: 2024-09-03T11:52:03Z|00010|acl_log(ovn_pinctrl0)|INFO|name="neutron-63935819-1c6a-4fb8-b765-338d98734e7e", verdict=allow, severity=info, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=fa:16:3e:f5:c7:dd,dl_dst=fa:16:3e:cb:91:13,nw_src=192.168.122.10,nw_dst=10.100.0.10,nw_tos=0,nw_ecn=0,nw_ttl=61,nw_frag=no,icmp_type=8,icmp_code=0
      /var/log/messages:118810:Sep  3 07:52:03 np0001998469 ovn_controller[66565]: 2024-09-03T11:52:03Z|00011|acl_log(ovn_pinctrl0)|INFO|name="neutron-63935819-1c6a-4fb8-b765-338d98734e7e", verdict=allow, severity=info, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=fa:16:3e:cb:91:13,dl_dst=fa:16:3e:f5:c7:dd,nw_src=10.100.0.10,nw_dst=192.168.122.10,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=0,icmp_code=0
      /var/log/secure:3324:Sep  3 07:52:03 np0001998469 sudo[135748]:    zuul : PWD=/home/zuul ; USER=root ; COMMAND=/bin/grep acl_log /var/log/containers/openvswitch/ovn-controller.log
      /var/log/secure:3330:Sep  3 07:57:22 np0001998469 sudo[135888]:    zuul : TTY=pts/0 ; PWD=/home/zuul ; USER=root ; COMMAND=/bin/grep --color=always -IErin acl_log /var/log/containers/
      /var/log/secure:3333:Sep  3 07:57:29 np0001998469 sudo[135892]:    zuul : TTY=pts/0 ; PWD=/home/zuul ; USER=root ; COMMAND=/bin/grep --color=always -IErin acl_log /var/log/
      

      [3]

      /var/log/containers/stdouts/ovn_controller.log
      /var/log/containers/openvswitch/ovn-controller.log
      /opt/stack/logs/ovn-controller.log

            Unassigned Unassigned
            rhn-support-mblue Maor Blaustein
            Maor Blaustein Maor Blaustein
            rhos-dfg-networking-squad-neutron
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: