Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-11192

Tests revolving around dedicated cpus are failing in whitebox after in place update

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • True
    • Hide

      waiting for a reproduction with logs

      Show
      waiting for a reproduction with logs
    • False
    • ?
    • None

      The tests revolving around dedicated cpus are failing in whitebox are failing post update .  Nova-scheduler indicates it cannot find candidates but the resource providers is showing available PCPUs and none are in use.  But the scheduler is saying it would over subscribe if it tried to place the guest. 

       

      [zuul@controller ~]$ oc rsh openstackclientsh-5.1$ openstack resource provider list+--------------------------------------+--------------------------------+------------+--------------------------------------+----------------------+| uuid                                 | name                           | generation | root_provider_uuid                   | parent_provider_uuid |+--------------------------------------+--------------------------------+------------+--------------------------------------+----------------------+| 611c310b-37ba-4e5f-87fe-916e753bf843 | compute-0.ctlplane.example.com |        485 | 611c310b-37ba-4e5f-87fe-916e753bf843 | None                 || 6eec71ed-299f-46d2-a4f8-4d6d34fa8220 | compute-1.ctlplane.example.com |        400 | 6eec71ed-299f-46d2-a4f8-4d6d34fa8220 | None                 |+--------------------------------------+--------------------------------+------------+--------------------------------------+----------------------+sh-5.1$ openstack resource provider inventory list 6eec71ed-299f-46d2-a4f8-4d6d34fa8220+----------------+------------------+----------+----------+----------+-----------+-------+------+| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |+----------------+------------------+----------+----------+----------+-----------+-------+------+| MEMORY_MB      |              1.0 |        1 |    15757 |      512 |         1 | 15757 |    0 || DISK_GB        |              0.9 |        1 |       13 |        1 |         1 |    13 |    0 || PCPU           |              1.0 |        1 |        4 |        0 |         1 |     4 |    0 || VCPU           |              4.0 |        1 |        2 |        0 |         1 |     2 |    0 |+----------------+------------------+----------+----------+----------+-----------+-------+------+sh-5.1$ openstack resource provider inventory list 611c310b-37ba-4e5f-87fe-916e753bf843+----------------+------------------+----------+----------+----------+-----------+-------+------+| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |+----------------+------------------+----------+----------+----------+-----------+-------+------+| MEMORY_MB      |              1.0 |        1 |    15757 |      512 |         1 | 15757 |    0 || DISK_GB        |              0.9 |        1 |       13 |        2 |         1 |    13 |    0 || PCPU           |              1.0 |        1 |        4 |        0 |         1 |     4 |    0 || VCPU           |              4.0 |        1 |        2 |        0 |         1 |     2 |    0 |+----------------+------------------+----------+----------+----------+-----------+-------+------+  nova-scheduler:2024-10-29 19:55:29.643 1 DEBUG nova.scheduler.filters [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] NUMATopologyFilter tries allocation candidate: {'allocations': {'611c310b-37ba-4e5f-87fe-916e753bf843': {'resources': {'DISK_GB': 1, 'MEMORY_MB': 256, 'PCPU': 1}}}, 'mappings': {'': ['611c310b-37ba-4e5f-87fe-916e753bf843']}} filter_candidates /usr/lib/python3.9/site-packages/nova/scheduler/filters/__init__.py:772024-10-29 19:55:29.643 1 DEBUG nova.virt.hardware [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([0])) on host_cell NUMACell(cpu_usage=0,cpuset=set([6]),id=1,memory=8016,memory_usage=0,mempages=[NUMAPagesTopology,NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pcpuset=set([]),pinned_cpus=set([]),siblings=[set([6])],socket=None) _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:9342024-10-29 19:55:29.643 1 DEBUG nova.virt.hardware [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] No specific pagesize requested for instance, selected pagesize: 4 _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:9632024-10-29 19:55:29.644 1 DEBUG nova.virt.hardware [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Not enough host cell CPUs to fit instance cell; required: 1 + 0 as overhead, actual: 0 _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:10012024-10-29 19:55:29.644 1 DEBUG nova.virt.hardware [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([0])) on host_cell NUMACell(cpu_usage=0,cpuset=set([2]),id=0,memory=7741,memory_usage=256,mempages=[NUMAPagesTopology,NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pcpuset=set([0]),pinned_cpus=set([0]),siblings=[set([0]),set([2])],socket=None) _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:9342024-10-29 19:55:29.644 1 DEBUG nova.virt.hardware [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] No specific pagesize requested for instance, selected pagesize: 4 _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:9632024-10-29 19:55:29.644 1 DEBUG nova.virt.hardware [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Instance has requested pinned CPUs _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:10282024-10-29 19:55:29.644 1 DEBUG nova.virt.hardware [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Not enough available CPUs to schedule instance. Oversubscription is not possible with pinned instances. Required: 1 (1 + 0), actual: 0 _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:10312024-10-29 19:55:29.644 1 DEBUG nova.scheduler.filters [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] NUMATopologyFilter rejected allocation candidate: {'allocations': {'611c310b-37ba-4e5f-87fe-916e753bf843': {'resources': {'DISK_GB': 1, 'MEMORY_MB': 256, 'PCPU': 1}}}, 'mappings': {'': ['611c310b-37ba-4e5f-87fe-916e753bf843']}} filter_candidates /usr/lib/python3.9/site-packages/nova/scheduler/filters/__init__.py:882024-10-29 19:55:29.645 1 DEBUG nova.scheduler.filters.numa_topology_filter [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] [instance: b4385854-40c8-43dc-8b9b-24ffc47c8c9a] compute-0.ctlplane.example.com, compute-0.ctlplane.example.com fails NUMA topology requirements. The instance does not fit on this host. host_passes /usr/lib/python3.9/site-packages/nova/scheduler/filters/numa_topology_filter.py:1162024-10-29 19:55:29.645 1 INFO nova.filters [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Filter NUMATopologyFilter returned 0 hosts2024-10-29 19:55:29.645 1 DEBUG nova.filters [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Filtering removed all hosts for the request with instance ID 'b4385854-40c8-43dc-8b9b-24ffc47c8c9a'. Filter results: [('AggregateInstanceExtraSpecsFilter', [('compute-1.ctlplane.example.com', 'compute-1.ctlplane.example.com'), ('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('ComputeFilter', [('compute-1.ctlplane.example.com', 'compute-1.ctlplane.example.com'), ('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('ComputeCapabilitiesFilter', [('compute-1.ctlplane.example.com', 'compute-1.ctlplane.example.com'), ('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('ImagePropertiesFilter', [('compute-1.ctlplane.example.com', 'compute-1.ctlplane.example.com'), ('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('ServerGroupAntiAffinityFilter', [('compute-1.ctlplane.example.com', 'compute-1.ctlplane.example.com'), ('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('ServerGroupAffinityFilter', [('compute-1.ctlplane.example.com', 'compute-1.ctlplane.example.com'), ('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('SameHostFilter', [('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('DifferentHostFilter', [('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('PciPassthroughFilter', [('compute-0.ctlplane.example.com', 'compute-0.ctlplane.example.com')]), ('NUMATopologyFilter', None)] get_filtered_objects /usr/lib/python3.9/site-packages/nova/filters.py:1142024-10-29 19:55:29.645 1 INFO nova.filters [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Filtering removed all hosts for the request with instance ID 'b4385854-40c8-43dc-8b9b-24ffc47c8c9a'. Filter results: ['AggregateInstanceExtraSpecsFilter: (start: 2, end: 2)', 'ComputeFilter: (start: 2, end: 2)', 'ComputeCapabilitiesFilter: (start: 2, end: 2)', 'ImagePropertiesFilter: (start: 2, end: 2)', 'ServerGroupAntiAffinityFilter: (start: 2, end: 2)', 'ServerGroupAffinityFilter: (start: 2, end: 2)', 'SameHostFilter: (start: 2, end: 1)', 'DifferentHostFilter: (start: 1, end: 1)', 'PciPassthroughFilter: (start: 1, end: 1)', 'NUMATopologyFilter: (start: 1, end: 0)']2024-10-29 19:55:29.645 1 DEBUG nova.scheduler.manager [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] Filtered [] _get_sorted_hosts /usr/lib/python3.9/site-packages/nova/scheduler/manager.py:6752024-10-29 19:55:29.645 1 DEBUG nova.scheduler.manager [None req-30eb1d4c-60d0-4b4e-a04c-26cbc2a278d3 cdb5cb13a0db4b6ea15ccc3b14d9d3a1 ab5ddd8288a64d93aee53c342bdeb484 - - default default] There are 0 hosts available but 1 instances requested to build. _ensure_sufficient_hosts /usr/lib/python3.9/site-packages/nova/scheduler/manager.py:494 

      Job link : https://gitlab.cee.redhat.com/ci-framework/ci-framework-testproject/-/merge_requests/794 
      Job logs : https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/zuul/t/components-integration/build/45bc3c31487846c680960a94663d1298

      Whitebox Tests results pre update : 

      https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/94/794/56d85c67bfd5ab7c1a552fe578d2afc77441417b/check-gitlab-cee/component-compute-edpm-update-rhel9-rhoso18.0-crc-local-storage/45bc3c3/controller/ci-framework-data/tests/test_operator/tempest-tests-tempest-whitebox-workflow-step-1/

      Whitebox Test results post-update :

      https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/94/794/56d85c67bfd5ab7c1a552fe578d2afc77441417b/check-gitlab-cee/component-compute-edpm-update-rhel9-rhoso18.0-crc-local-storage/45bc3c3/controller/ci-framework-data/tests/test_operator_update/post-update-tempest-tests-tempest-whitebox-workflow-step-1/stestr_results.html

      Containers (pre and post update) :
      https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/94/794/56d85c67bfd5ab7c1a552fe578d2afc77441417b/check-gitlab-cee/component-compute-edpm-update-rhel9-rhoso18.0-crc-local-storage/45bc3c3/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/crs/openstackcontrolplanes.core.openstack.org/controlplane.yaml

      Before update repos : https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/94/794/56d85c67bfd5ab7c1a552fe578d2afc77441417b/check-gitlab-cee/component-compute-edpm-update-rhel9-rhoso18.0-crc-local-storage/45bc3c3/controller/ci-framework-data/artifacts/before_update_repos/

              rh-ee-bgibizer Balazs Gibizer
              ashigupt@redhat.com Ashish Gupta
              rhos-workloads-compute
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: