Uploaded image for project: 'OpenStack as Infra'
  1. OpenStack as Infra
  2. OSASINFRA-2799

"Length too long" Kuryr upstream CI issue

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Cannot Reproduce
    • Icon: Undefined Undefined
    • None
    • None
    • Kuryr
    • None
    • False
    • None
    • False

      Symptoms
      Despite all tests passing, `tempest` command ends up with a failure and similar message:

      2022-03-07 18:41:14.784763 | controller |

      {0} kuryr_tempest_plugin.tests.scenario.test_service.TestDeployment.test_deployment_scale [1190.505081s] ... ok
      2022-03-07 18:50:10.047628 | controller | {3} kuryr_tempest_plugin.tests.scenario.test_service.TestServiceScenario.test_pod_service_curl [63.978711s] ... ok
      2022-03-07 18:51:30.708187 | controller | {2} kuryr_tempest_plugin.tests.scenario.test_service.TestServiceWithoutSelectorScenario.test_service_without_selector [903.504235s] ... ok
      2022-03-07 18:55:06.847091 | controller | {0}

      kuryr_tempest_plugin.tests.scenario.test_service.TestListenerTimeoutScenario.test_updated_listener_timeouts [803.924965s] ... ok
      2022-03-07 19:11:12.691250 | controller |

      {0}

      kuryr_tempest_plugin.tests.scenario.test_service.TestUdpServiceScenario.test_service_udp_ping [879.442654s] ... ok
      2022-03-07 19:12:53.973101 | controller | Length too long: 4278817
      2022-03-07 19:12:56.099827 | controller | ERROR: InvocationError for command /opt/stack/tempest/.tox/tempest/bin/tempest run --regex '^(kuryr_tempest_plugin.tests.)' --concurrency=4 (exited with code 1)
      2022-03-07 19:12:56.220687 | controller | ___________________________________ summary ____________________________________
      2022-03-07 19:12:56.222268 | controller | ERROR: all: commands failed
      2022-03-07 19:12:58.157773 | controller | ERROR
      2022-03-07 19:12:58.158097 | controller |

      { 2022-03-07 19:12:58.158181 | controller | "delta": "0:52:08.469716", 2022-03-07 19:12:58.158253 | controller | "end": "2022-03-07 19:12:56.951862", 2022-03-07 19:12:58.158322 | controller | "msg": "non-zero return code", 2022-03-07 19:12:58.158395 | controller | "rc": 1, 2022-03-07 19:12:58.158462 | controller | "start": "2022-03-07 18:20:48.482146" 2022-03-07 19:12:58.158527 | controller | }

      Investigations
      This is because tests log too much. And they log too much because each K8s API call is logged.
      Root cause
      See above.
      Solution
      Mitigation: https://review.opendev.org/c/openstack/kuryr-tempest-plugin/+/830384
      Brute force: https://review.opendev.org/c/openstack/kuryr-tempest-plugin/+/830842

              Unassigned Unassigned
              mdulko MichaƂ Dulko (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: