-
Bug
-
Resolution: Done
-
Normal
-
None
-
None
-
False
-
-
False
-
?
-
?
-
?
-
?
-
None
-
-
-
Moderate
A CI job failed like this:
TASK [cinder_adoption : Pause to allow backend drivers to start] *************** skipping: [localhost] => {"changed": false, "false_condition": "cinder_volume_backend | default('') != '' or cinder_backup_backend | default('') != ''", "skip_reason": "Conditional result was False"} TASK [cinder_adoption : check that Cinder is reachable and its endpoints are defined] *** FAILED - RETRYING: [localhost]: check that Cinder is reachable and its endpoints are defined (60 retries left). changed: [localhost] => {"attempts": 2, "changed": true, "cmd": "set -euxo pipefail\n\n\nalias openstack=\"oc exec -t openstackclient -- openstack\"\n\n${BASH_ALIASES[openstack]} endpoint list | grep cinder\n${BASH_ALIASES[openstack]} volume type list\n", "delta": "0:00:04.802908", "end": "2024-07-19 08:34:14.065022", "msg": "", "rc": 0, "start": "2024-07-19 08:34:09.262114", "stderr": "+ alias 'openstack=oc exec -t openstackclient -- openstack'\n+ oc exec -t openstackclient -- openstack endpoint list\n+ grep cinder\n+ oc exec -t openstackclient -- openstack volume type list", "stderr_lines": ["+ alias 'openstack=oc exec -t openstackclient -- openstack'", "+ oc exec -t openstackclient -- openstack endpoint list", "+ grep cinder", "+ oc exec -t openstackclient -- openstack volume type list"], "stdout": "| 9dae5dbb596c445ea4ff1d5f1f3a36c7 | regionOne | cinderv3 | volumev3 | True | internal | https://cinder-internal.openstack.svc:8776/v3 |\n| bc9a52830c5a414a80567e6fbea6d121 | regionOne | cinderv3 | volumev3 | True | public | https://cinder-public-openstack.apps-crc.testing/v3 |\n+--------------------------------------+---------+-----------+\n| ID | Name | Is Public |\n+--------------------------------------+---------+-----------+\n| 778c7611-609b-4e81-8cd8-2d03a49f79d9 | tripleo | True |\n+--------------------------------------+---------+-----------+", "stdout_lines": ["| 9dae5dbb596c445ea4ff1d5f1f3a36c7 | regionOne | cinderv3 | volumev3 | True | internal | https://cinder-internal.openstack.svc:8776/v3 |", "| bc9a52830c5a414a80567e6fbea6d121 | regionOne | cinderv3 | volumev3 | True | public | https://cinder-public-openstack.apps-crc.testing/v3 |", "+--------------------------------------+---------+-----------+", "| ID | Name | Is Public |", "+--------------------------------------+---------+-----------+", "| 778c7611-609b-4e81-8cd8-2d03a49f79d9 | tripleo | True |", "+--------------------------------------+---------+-----------+"]} TASK [cinder_adoption : wait for Cinder volume to be up and ready] ************* skipping: [localhost] => {"changed": false, "false_condition": "cinder_volume_backend != ''", "skip_reason": "Conditional result was False"} TASK [cinder_adoption : wait for Cinder backup to be up and ready] ************* skipping: [localhost] => {"changed": false, "false_condition": "cinder_backup_backend != ''", "skip_reason": "Conditional result was False"} TASK [cinder_adoption : Cinder online data migrations] ************************* fatal: [localhost]: FAILED! => {"changed": true, "cmd": "set -euxo pipefail\n\n\noc exec -it cinder-scheduler-0 -- cinder-manage db online_data_migrations\n", "delta": "0:00:00.166316", "end": "2024-07-19 08:34:14.485782", "msg": "non-zero return code", "rc": 1, "start": "2024-07-19 08:34:14.319466", "stderr": "+ oc exec -it cinder-scheduler-0 -- cinder-manage db online_data_migrations\nDefaulted container \"cinder-scheduler\" out of: cinder-scheduler, probe\nUnable to use a TTY - input is not a terminal or the right kind of file\nerror: unable to upgrade connection: container not found (\"cinder-scheduler\")", "stderr_lines": ["+ oc exec -it cinder-scheduler-0 -- cinder-manage db online_data_migrations", "Defaulted container \"cinder-scheduler\" out of: cinder-scheduler, probe", "Unable to use a TTY - input is not a terminal or the right kind of file", "error: unable to upgrade connection: container not found (\"cinder-scheduler\")"], "stdout": "", "stdout_lines": []}
But then looking at must-gather data, the cinder-scheduler pod was running fine:
pod/cinder-2f0e-account-create-bdkwv 0/1 Completed 0 3m8s pod/cinder-api-0 2/2 Running 0 2m39s pod/cinder-db-create-dsnwq 0/1 Completed 0 3m18s pod/cinder-db-sync-cfpvx 0/1 Completed 0 3m2s pod/cinder-scheduler-0 2/2 Running 0 2m25s
I think we just need a wait for the scheduler to come up before we attempt the online data migrations.