Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: rhos-18.0.0
Affects Version/s: rhos-18.0.0
Component/s: data-plane-adoption, internal
Labels:
None

Epic Link:
As an Adoption team member, I want automation to deploy OpenStack 17.1 with control plane on RHEV that can be networked together with OpenShift, so that I can test adopting that deployment into podified 18.0.
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Dev Approval:
?
Docs Approval:
?
Feature Link:
OSPRH-3714 - Red Hat OpenStack 18.0 Data Plane Adoption (Post OSP18 GA)
PM Approval:
?
QE Approval:
?
Regression:
None
Intelligence Requested:
Market:

Severity:
Important

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

ovn adoption fails on 3 node openshift setup on RHV envs as

TASK [ovn_adoption : deploy podified OVN ovsdb-servers] ************************
Monday 22 July 2024  00:22:09 +0000 (0:00:01.675)       0:09:32.505 *********** 
changed: [localhost] => {"changed": true, "cmd": "set -euxo pipefail\n\nexport KUBECONFIG=~/adoption/kubeconfig\n\noc patch openstackcontrolplane openstack --type=merge --patch '\nspec:\n  ovn:\n    enabled: true\n    template:\n      ovnDBCluster:\n        ovndbcluster-nb:\n          dbType: NB\n          storageRequest: 10G\n          networkAttachment: internalapi\n        ovndbcluster-sb:\n          dbType: SB\n          storageRequest: 10G\n          networkAttachment: internalapi\n'\n", "delta": "0:00:00.313113", "end": "2024-07-22 00:22:10.251884", "msg": "", "rc": 0, "start": "2024-07-22 00:22:09.938771", "stderr": "+ export KUBECONFIG=/home/stack/adoption/kubeconfig\n+ KUBECONFIG=/home/stack/adoption/kubeconfig\n+ oc patch openstackcontrolplane openstack --type=merge --patch '\nspec:\n  ovn:\n    enabled: true\n    template:\n      ovnDBCluster:\n        ovndbcluster-nb:\n          dbType: NB\n          storageRequest: 10G\n          networkAttachment: internalapi\n        ovndbcluster-sb:\n          dbType: SB\n          storageRequest: 10G\n          networkAttachment: internalapi\n'", "stderr_lines": ["+ export KUBECONFIG=/home/stack/adoption/kubeconfig", "+ KUBECONFIG=/home/stack/adoption/kubeconfig", "+ oc patch openstackcontrolplane openstack --type=merge --patch '", "spec:", "  ovn:", "    enabled: true", "    template:", "      ovnDBCluster:", "        ovndbcluster-nb:", "          dbType: NB", "          storageRequest: 10G", "          networkAttachment: internalapi", "        ovndbcluster-sb:", "          dbType: SB", "          storageRequest: 10G", "          networkAttachment: internalapi", "'"], "stdout": "openstackcontrolplane.core.openstack.org/openstack patched", "stdout_lines": ["openstackcontrolplane.core.openstack.org/openstack patched"]}

TASK [ovn_adoption : wait for OVN ovsdb-servers to start up] *******************
Monday 22 July 2024  00:22:10 +0000 (0:00:00.629)       0:09:33.134 *********** 
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (60 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (59 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (58 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (57 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (56 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (55 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (54 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (53 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (52 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (51 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (50 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (49 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (48 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (47 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (46 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (45 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (44 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (43 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (42 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (41 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (40 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (39 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (38 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (37 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (36 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (35 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (34 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (33 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (32 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (31 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (30 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (29 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (28 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (27 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (26 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (25 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (24 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (23 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (22 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (21 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (20 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (19 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (18 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (17 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (16 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (15 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (14 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (13 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (12 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (11 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (10 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (9 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (8 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (7 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (6 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (5 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (4 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (3 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (2 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 60, "changed": true, "cmd": "set -euxo pipefail\n\nexport KUBECONFIG=~/adoption/kubeconfig\n\noc get pod --selector=service=ovsdbserver-nb -o jsonpath='{.items[0].status.phase}{\"\\n\"}' | grep Running\noc get pod --selector=service=ovsdbserver-sb -o jsonpath='{.items[0].status.phase}{\"\\n\"}' | grep Running\n", "delta": "0:00:00.202294", "end": "2024-07-22 00:24:39.530875", "msg": "non-zero return code", "rc": 1, "start": "2024-07-22 00:24:39.328581", "stderr": "+ export KUBECONFIG=/home/stack/adoption/kubeconfig\n+ KUBECONFIG=/home/stack/adoption/kubeconfig\n+ oc get pod --selector=service=ovsdbserver-nb -o 'jsonpath={.items[0].status.phase}{\"\\n\"}'\n+ grep Running\nerror: error executing jsonpath \"{.items[0].status.phase}{\\\"\\\\n\\\"}\": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:\n\ttemplate was:\n\t\t{.items[0].status.phase}{\"\\n\"}\n\tobject given to jsonpath engine was:\n\t\tmap[string]interface {}{\"apiVersion\":\"v1\", \"items\":[]interface {}{}, \"kind\":\"List\", \"metadata\":map[string]interface {}{\"resourceVersion\":\"\"}}", "stderr_lines": ["+ export KUBECONFIG=/home/stack/adoption/kubeconfig", "+ KUBECONFIG=/home/stack/adoption/kubeconfig", "+ oc get pod --selector=service=ovsdbserver-nb -o 'jsonpath={.items[0].status.phase}{\"\\n\"}'", "+ grep Running", "error: error executing jsonpath \"{.items[0].status.phase}{\\\"\\\\n\\\"}\": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:", "\ttemplate was:", "\t\t{.items[0].status.phase}{\"\\n\"}", "\tobject given to jsonpath engine was:", "\t\tmap[string]interface {}{\"apiVersion\":\"v1\", \"items\":[]interface {}{}, \"kind\":\"List\", \"metadata\":map[string]interface {}{\"resourceVersion\":\"\"}}"], "stdout": "", "stdout_lines": []}
TASK [ovn_adoption : deploy podified OVN ovsdb-servers] ************************
Monday 22 July 2024  00:22:09 +0000 (0:00:01.675)       0:09:32.505 *********** 
changed: [localhost] => {"changed": true, "cmd": "set -euxo pipefail\n\nexport KUBECONFIG=~/adoption/kubeconfig\n\noc patch openstackcontrolplane openstack --type=merge --patch '\nspec:\n  ovn:\n    enabled: true\n    template:\n      ovnDBCluster:\n        ovndbcluster-nb:\n          dbType: NB\n          storageRequest: 10G\n          networkAttachment: internalapi\n        ovndbcluster-sb:\n          dbType: SB\n          storageRequest: 10G\n          networkAttachment: internalapi\n'\n", "delta": "0:00:00.313113", "end": "2024-07-22 00:22:10.251884", "msg": "", "rc": 0, "start": "2024-07-22 00:22:09.938771", "stderr": "+ export KUBECONFIG=/home/stack/adoption/kubeconfig\n+ KUBECONFIG=/home/stack/adoption/kubeconfig\n+ oc patch openstackcontrolplane openstack --type=merge --patch '\nspec:\n  ovn:\n    enabled: true\n    template:\n      ovnDBCluster:\n        ovndbcluster-nb:\n          dbType: NB\n          storageRequest: 10G\n          networkAttachment: internalapi\n        ovndbcluster-sb:\n          dbType: SB\n          storageRequest: 10G\n          networkAttachment: internalapi\n'", "stderr_lines": ["+ export KUBECONFIG=/home/stack/adoption/kubeconfig", "+ KUBECONFIG=/home/stack/adoption/kubeconfig", "+ oc patch openstackcontrolplane openstack --type=merge --patch '", "spec:", "  ovn:", "    enabled: true", "    template:", "      ovnDBCluster:", "        ovndbcluster-nb:", "          dbType: NB", "          storageRequest: 10G", "          networkAttachment: internalapi", "        ovndbcluster-sb:", "          dbType: SB", "          storageRequest: 10G", "          networkAttachment: internalapi", "'"], "stdout": "openstackcontrolplane.core.openstack.org/openstack patched", "stdout_lines": ["openstackcontrolplane.core.openstack.org/openstack patched"]}

TASK [ovn_adoption : wait for OVN ovsdb-servers to start up] *******************
Monday 22 July 2024  00:22:10 +0000 (0:00:00.629)       0:09:33.134 *********** 
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (60 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (59 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (58 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (57 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (56 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (55 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (54 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (53 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (52 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (51 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (50 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (49 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (48 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (47 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (46 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (45 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (44 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (43 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (42 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (41 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (40 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (39 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (38 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (37 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (36 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (35 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (34 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (33 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (32 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (31 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (30 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (29 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (28 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (27 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (26 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (25 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (24 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (23 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (22 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (21 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (20 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (19 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (18 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (17 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (16 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (15 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (14 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (13 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (12 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (11 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (10 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (9 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (8 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (7 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (6 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (5 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (4 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (3 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (2 retries left).
FAILED - RETRYING: [localhost]: wait for OVN ovsdb-servers to start up (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 60, "changed": true, "cmd": "set -euxo pipefail\n\nexport KUBECONFIG=~/adoption/kubeconfig\n\noc get pod --selector=service=ovsdbserver-nb -o jsonpath='{.items[0].status.phase}{\"\\n\"}' | grep Running\noc get pod --selector=service=ovsdbserver-sb -o jsonpath='{.items[0].status.phase}{\"\\n\"}' | grep Running\n", "delta": "0:00:00.202294", "end": "2024-07-22 00:24:39.530875", "msg": "non-zero return code", "rc": 1, "start": "2024-07-22 00:24:39.328581", "stderr": "+ export KUBECONFIG=/home/stack/adoption/kubeconfig\n+ KUBECONFIG=/home/stack/adoption/kubeconfig\n+ oc get pod --selector=service=ovsdbserver-nb -o 'jsonpath={.items[0].status.phase}{\"\\n\"}'\n+ grep Running\nerror: error executing jsonpath \"{.items[0].status.phase}{\\\"\\\\n\\\"}\": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:\n\ttemplate was:\n\t\t{.items[0].status.phase}{\"\\n\"}\n\tobject given to jsonpath engine was:\n\t\tmap[string]interface {}{\"apiVersion\":\"v1\", \"items\":[]interface {}{}, \"kind\":\"List\", \"metadata\":map[string]interface {}{\"resourceVersion\":\"\"}}", "stderr_lines": ["+ export KUBECONFIG=/home/stack/adoption/kubeconfig", "+ KUBECONFIG=/home/stack/adoption/kubeconfig", "+ oc get pod --selector=service=ovsdbserver-nb -o 'jsonpath={.items[0].status.phase}{\"\\n\"}'", "+ grep Running", "error: error executing jsonpath \"{.items[0].status.phase}{\\\"\\\\n\\\"}\": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:", "\ttemplate was:", "\t\t{.items[0].status.phase}{\"\\n\"}", "\tobject given to jsonpath engine was:", "\t\tmap[string]interface {}{\"apiVersion\":\"v1\", \"items\":[]interface {}{}, \"kind\":\"List\", \"metadata\":map[string]interface {}{\"resourceVersion\":\"\"}}"], "stdout": "", "stdout_lines": []}

Full log output is https://sf.hosted.upshift.rdu2.redhat.com/logs/73/473/6b4e0e65d352f3e3d972f24c1013fc6b7d569be9/check-gitlab-cee/rhev-rhel9-osp18-base-adoption-test-archana/c079d24/undercloud/data-plane-adoption-tests-repo/data-plane-adoption/tests/logs/test_minimal_out_2024-07-22T00:12:36UTC.log
Job logs: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-rhev-rhel9-rhos-18-adoption-internal

Assignee:: Sergii Golovatiuk

Reporter:: Sergii Golovatiuk

Team:: rhos-dfg-upgrades-squad-upgrades

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2024/07/22 12:10 PM

Updated:: 2024/07/31 8:33 AM

Resolved:: 2024/07/31 8:33 AM

Details

Description

Attachments

Activity

People

Dates

PagerDuty