-
Epic
-
Resolution: Done
-
Blocker
-
None
-
None
-
Fix gitlab-runner installation on f39 runners to unblock osbuild CI
-
Done
-
image-builder-1
-
0% To Do, 0% In Progress, 100% Done
-
False
Problem:
- osbuild CI started to fail consistently when bootstrapping the gitlab runner on F39 image. E.g. https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/pipelines/1606107612 (https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/jobs/8739935228)
Investigation:
- It turns out that the latest gitlab-runner RPM depends on gitlab-runner-helper-images RPM. Due to this reason, manually installing the runner RPM (as used in https://github.com/osbuild/gitlab-ci-terraform-executor/blob/a44f3362804de00305f93a12a4feebcf96b23eb6/prepare#L127) stopped working.
- The installation fails on all of our gitlab runner VMs, but some images already have the runner installed from a repository (e.g https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/jobs/8739935231#L1915). Specifically those that were built by the https://gitlab.cee.redhat.com/osbuild/ci-image-refresh-bot/. The bot uses Gitlab runner repository to install the package (https://gitlab.cee.redhat.com/osbuild/ci-image-refresh-bot/-/blob/main/build_matrix.py?ref_type=heads#L139), instead of manually installing the RPM. The repo contains all dependencies, thus they are puled.
- The F39 runner image is a Fedora-Cloud-Base-39-20231204.0 image; therefore, it does not come with the gitlab-runner RPM pre-installed, and the installation by the executor prep script fails.
Goal:
- Unblock osbuild CI, which started to fail consistently when bootstrapping the gitlab runner on F39 image. E.g. https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/pipelines/1606107612 (https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/jobs/8739935228)
- Fix the problem in all appropriate places
Acceptance Criteria:
- Quick fix:
- Update F39 AWS runner image to the one rebuilt by the ci-image-refresh-bot, which has the gitlab-runner pre-installed.
- Update the terraform ref in osbuild repository.
- Proper fix:
- Modify the https://github.com/osbuild/gitlab-ci-terraform-executor/blob/main/prepare script to install gitlab-runner as described in https://docs.gitlab.com/runner/install/linux-manually.html.
