Critical One of our customer is requesting a reasonable use case for ansible-builder enhancement is as follows:
Scenario:
A custom AAP execution environment is required with the following requirements
- Using Private Automation Hub as container registry with Ansible EE Base Image and EE Builder are synced into Private Automation Hub from Red Hat Registry
- Collections are added from Private Automation Hub
- Base OS Server running ansible-builder is subscribed to Satellite/Capsule with GPG checking enabled for third party repositories
- Base OS Server is behind proxy and has custom CA Signed Certificates
- Private Automation Hub also uses custom CA-signed Certificates
- ansible-builder prepend and append parameters only applies to "FROM $EE_BASE_IMAGE" part of build and not to all build phase
- any modification done to context/Container file will just be overwritten (and removed) when ansible-builder build is triggered.
Expected Outcome:
- ansible-builder to allow custom injections/arguments (prepend/append) in ContainerFile to "FROM $EE_BASE_IMAGE as galaxy" and "FROM $EE_BUILDER_IMAGE as builder"
=================================================================================================================================
Actual Results:
Scenario 1: Using command "ansible-builder build -t test_ee -v 3"
Error 1: Cannot download collections from Private Automation Hub due to SSL Certificate Verification Errors
Error in [1/3] STEP 8/8: RUN ANSIBLE_GALAXY_DISABLE_GPG_VERIFY=1 ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path "/usr/share/ansible/collections"
ERROR! Unknown error when attempting to call Galaxy at '
https://ansible-hub-dev.nz.thenational.com/api/galaxy/content/community/api
': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)>
Error: error building at STEP "RUN ANSIBLE_GALAXY_DISABLE_GPG_VERIFY=1 ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path "/usr/share/ansible/collections"": error while running runtime: exit status 1
Workaround:
--> add ANSIBLE_GALAXY_CLI_COLLECTIONS_OPTS: "-c" in execution-environment.yml which is kinda insecure
OR
--> create ContainerFile manually via, "ansible-builder create" then modify generated context/ContainerFile and the custom CA Certificates under FROM$EE_BASE_IMAGE as galaxy side like this below, then resume to build container via podman build -f context/ContainerFile -t <tagname>
FROM $EE_BASE_IMAGE as galaxy
ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=
USER root
COPY <custom-CA-cert> /etc/pki/ca-trust/source/anchors/
RUN update-ca-trust
Expected ansible-builder enhancement:
- that ansible-builder can allow custom injections to be able to add CA Certs in build stage --> FROM $EE_BASE_IMAGE as galaxy as above
Error 2: $EE_BUILDER_IMAGE as builder (ansible-builder-rhel8 container) cannot reach UBI repositories
Downloading metadata...
error: cannot update repo 'ubi-8-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Curl error (28): Timeout was reached for
https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/x86_64/baseos/os/repodata/repomd.xml
[Connection timed out after 30001 milliseconds]
Error: error building at STEP "RUN assemble": error while running runtime: exit status 1
Workaround:
Modify ContainerFile and add proxy in EE_BUILDER_IMAGE as builder then continue build via podman.
FROM $EE_BUILDER_IMAGE as builder
ENV HTTP_PROXY="insert proxy here"
ENV HTTPS_PROXY="insert proxy here"
Expected ansible-builder enhancement:
- that ansible-builder can allow custom injections to be able to add Proxy in build stage --> EE_BUILDER_IMAGE as builder
Error 3: Error installing packages in builder image due to GPG Checks
Downloading packages...
error: package python3-pathspec-0.6.0-1.el8.noarch cannot be verified and repo org_BNZ_EPEL_Restricted_epel_el8_x86_64_UnRestricted is GPG enabled: /var/cache/yum/metadata/org_BNZ_EPEL_Restricted_epel_el8_x86_64_UnRestricted-8-x86_64/packages/python3-pathspec-0.6.0-1.el8.noarch.rpm could not be verified.
/var/cache/yum/metadata/org_BNZ_EPEL_Restricted_epel_el8_x86_64_UnRestricted-8-x86_64/packages/python3-pathspec-0.6.0-1.el8.noarch.rpm: digest: SIGNATURE: NOT OK
Error: error building at STEP "RUN /output/install-from-bindep && rm -rf /output/wheels": error while running runtime: exit status 1
Workaround:
Modify ContainerFile and add GPGKeys to EE Builder Image then continue build via podman.
FROM $EE_BUILDER_IMAGE as builder
ENV HTTP_PROXY="insert proxy here"
ENV HTTPS_PROXY="insert proxy here"
COPY gpgkey /etc/pki/rpm-gpg
Expected ansible-builder enhancement:
- that ansible-builder can allow custom injections to be able to add GPG Keys in build stage --> EE_BUILDER_IMAGE as builder
==============
Without these workarounds, building custom execution environments is not possible (and easy?) when behind proxy, using custom CA Certificates and having third party gpg checks enabled repositories in Satellite.
Similar issues reported on ansible-builder github page:
https://github.com/ansible/ansible-builder/issues/350