Description of problem:

When trying to run make binaries on my system, compiling the MCD fails with the following error:

# github.com/openshift/machine-config-operator/cmd/machine-config-daemon
/usr/lib/golang/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/gcc -m64 -s -Wl,--build-id=0x1fcd95952cce501a25e4112db90c5471ec4a059a -o $WORK/b001/exe/a.out -Wl,--export-dynamic-symbol=_cgo_panic -Wl,--export-dynamic-symbol=_cgo_topofstack -Wl,--export-dynamic-symbol=crosscall2 -Wl,--export-dynamic-symbol=gogpgme_assuan_data_callback -Wl,--export-dynamic-symbol=gogpgme_assuan_inquiry_callback -Wl,--export-dynamic-symbol=gogpgme_assuan_status_callback -Wl,--export-dynamic-symbol=gogpgme_passfunc -Wl,--export-dynamic-symbol=gogpgme_readfunc -Wl,--export-dynamic-symbol=gogpgme_seekfunc -Wl,--export-dynamic-symbol=gogpgme_writefunc -Wl,--compress-debug-sections=zlib /tmp/go-link-2207548112/go.o /tmp/go-link-2207548112/000000.o /tmp/go-link-2207548112/000001.o /tmp/go-link-2207548112/000002.o /tmp/go-link-2207548112/000003.o /tmp/go-link-2207548112/000004.o /tmp/go-link-2207548112/000005.o /tmp/go-link-2207548112/000006.o /tmp/go-link-2207548112/000007.o /tmp/go-link-2207548112/000008.o /tmp/go-link-2207548112/000009.o /tmp/go-link-2207548112/000010.o /tmp/go-link-2207548112/000011.o /tmp/go-link-2207548112/000012.o /tmp/go-link-2207548112/000013.o /tmp/go-link-2207548112/000014.o /tmp/go-link-2207548112/000015.o /tmp/go-link-2207548112/000016.o /tmp/go-link-2207548112/000017.o /tmp/go-link-2207548112/000018.o /tmp/go-link-2207548112/000019.o /tmp/go-link-2207548112/000020.o /tmp/go-link-2207548112/000021.o /tmp/go-link-2207548112/000022.o /tmp/go-link-2207548112/000023.o /tmp/go-link-2207548112/000024.o /tmp/go-link-2207548112/000025.o /tmp/go-link-2207548112/000026.o /tmp/go-link-2207548112/000027.o /tmp/go-link-2207548112/000028.o /tmp/go-link-2207548112/000029.o /tmp/go-link-2207548112/000030.o /tmp/go-link-2207548112/000031.o -O2 -g -O2 -g -lresolv -O2 -g -lpthread -O2 -g -lgpgme -O2 -g -no-pie
/usr/bin/ld: cannot find -lgpgme: No such file or directory
collect2: error: ld returned 1 exit status

This can be easily reproduced in a clone of the MCO repo:

$ podman pull docker.io/library/golang:latest
$ podman run -it --rm -v "$PWD:/go/src/github.com/openshift/machine-config-operator:z" -w /go/src/github.com/openshift/machine-config-operator docker.io/library/golang:latest make binaries

Analysis:

On my system, I know that the proximate cause is that the gpgme-devel package is missing from my environment (I use customized Toolbox images and could easily add the missing library). However, looking at the hack/build-go.sh (https://github.com/openshift/machine-config-operator/blob/a43523ef4c49e31e74e9ce9dec1c5178da666984/hack/build-go.sh#L40-L42) script, we conditionally set Go tags when building the machine-config-controller. Modifying this conditional to include the MCD resolves the compilation error.

We have this dependency because of the github.com/containers/image/v5/signature package, which, as of this writing, is v5.35.0. Looking at that package, here is something that sticks out: https://github.com/containers/image/blob/617c288ea0420502677c6211b416b7d2dfecc67b/signature/mechanism_openpgp.go#L18-L25. In other words, the package, as well as the package it depends on (https://pkg.go.dev/golang.org/x/crypto/openpgp) make it clear that gpgme is the preferred implementation.

Additional considerations:

In CI, everything builds and runs because the builder images in our Dockerfile include the gpgme-devel library. However, all is not as it seems. Currently, the Machine Config Controller is built with the OpenPGP implementation whereas the Machine Config Daemon is built with the gpgme implementation. And the e2e test suite uses the OpenPGP implementation as well because of the Go tags. So, we should probably at least standardize on one of these implementations.

To prove this, I built the binaries using the OCP builder image that the MCO uses:

$ podman run -it --rm -v "$PWD:/go/src/github.com/openshift/machine-config-operator:z" -w /go/src/github.com/openshift/machine-config-operator registry.ci.openshift.org/ocp/builder:rhel-9-golang-1.24-openshift-4.20 make binaries

Next, I leveraged the Go toolchain to inspect the binaries that were built above and figure out what libraries they were built with:

$ go version -m _output/linux/amd64/machine-config-controller | grep -i -E '(pgp|gpg)'
        build   -tags=containers_image_openpgp,exclude_graphdriver_devicemapper,exclude_graphdriver_btrfs,containers_image_ostree_stub,strictfipsruntime

$ go version -m _output/linux/amd64/machine-config-daemon | grep -i -E '(pgp|gpg)'
        dep     github.com/proglottis/gpgme     v0.1.4

I also did the same thing against the three most popular projects that use the containers/image library, Podman, Buildah, and Skopeo:

$ go version -m /usr/bin/podman | grep -i -E '(pgp|gpg)'
        dep     github.com/proglottis/gpgme     v0.1.4
$ go version -m /usr/bin/buildah | grep -i -E '(pgp|gpg)'
        dep     github.com/proglottis/gpgme     v0.1.4
$ go version -m /usr/bin/skopeo | grep -i -E '(pgp|gpg)'
        dep     github.com/proglottis/gpgme     v0.1.4

All three binaries use the gpgme implementation, which supports the idea that we should be using it as well.

Proposed solution:

1. Ensure that all MCO developers have gpgme-devel on their machines. This can be easily obtained from DNF. Mac users will need to use Homebrew to install gpgme and pkgconf; I've verified that this works on my M3 MacBook Pro.
2. Remove the conditional GOTAG application in the Makefile and hack/build-go.sh files to ensure that is used in all cases. It's also worth pointing out that none of the other Go tags that we set are actually present in any of our vendored libraries, meaning that we could probably do away with them entirely.

Verification:

Assuming that all CI tests pass, we can verify the change by performing a build and using go version -m to verify that we're using the correct implementation.