While debugging https://issues.redhat.com/browse/OCPBUGS-18483, something that proved very useful was being able to attach pprof to a running MCO binary in an OpenShift cluster. For a basic background of what pprof is and how it's useful, check out this article: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/

When combined with something like $ oc port-forward , it is possible to attach to a running MCO binary in an OpenShift cluster and collect this information. This information can then be used to determine what portions of the MCO are consuming the most CPU and memory, which will allow us to make our code more efficient.

Adding the code itself is fairly straightforward:

import (
	_ "net/http/pprof"
)

// Within the Start() function (or anything else that runs for quite some time), we add something like this:
func Start(...) {
	go func() {
		klog.Infof("Launching server")
		klog.Infoln(http.ListenAndServe("localhost:6060", nil))
		klog.Infof("Launching server ended")
	}()
}

However, care must be taken because importing the net/http/pprof module introduces a side-effect. Specifically, it will modify the global default serving mux unless additional care is taken. The only other HTTP server I'm aware of within each MCO component is the Prometheus metrics endpoint. And taking a cursory glance at that code, it gets its own independent serving mux and does not rely on the global serving mux. That said, it could be worth taking the time to configure the pprof HTTP server to use its own dedicated serving mux.

Done When:

• All of the individual MCO components (machine-config-daemon, machine-config-server, etc.) have an optional CLI flag that will start a pprof HTTP server when used.
• Eventually, this should be wired up to a configurable knob to allow anyone to toggle this on or off though that is out-of-scope for this issue.