-
Bug
-
Resolution: Done
-
Major
-
1.8.0, 1.8.1, 1.8.2
-
2
-
False
-
-
False
-
-
Bug Fix
-
Done
-
-
-
RHDH Install 3285, RHDH Install 3286
Description of problem:
As reported in RHDHSUPP-305, errors from the 'skopeo copy' command executed by the install-dynamic-plugins.py script (when handling OCI plugins installation) are not surfaced anywhere in the logs. What's returned is only that the skopeo subprocess failed.
This makes it hard to troubleshoot, as the user had to manually exec into the container and run skopeo to figure out what the errors were.
Prerequisites (if any, like setup, operators/versions):
- Operator-based install
- RHDH 1.8
Steps to Reproduce
Configuration:
kind: ConfigMap apiVersion: v1 metadata: name: dynamic-plugins-rhdh data: dynamic-plugins.yaml: | includes: - dynamic-plugins.default.yaml plugins: # Intentional to make skopeo fail, but in the user's configuration, # they were pointing to a private registry and the TLS certificate information were missing - package: 'oci://fake-registry.example.com/my-org/my-plugin:1.2.3!some-plugin' disabled: false --- apiVersion: rhdh.redhat.com/v1alpha4 kind: Backstage metadata: name: rhdh spec: monitoring: enabled: true application: envs: - name: NODE_TLS_REJECT_UNAUTHORIZED value: "0" containers: - backstage-backend - install-dynamic-plugins dynamicPluginsConfigMapName: dynamic-plugins-rhdh
Actual results:
This is what we have in the init container logs: only the `skopeo command` exit code.
======= Skipping disabled dynamic plugin @redhat/backstage-plugin-scaffolder-backend-module-orchestrator-dynamic@1.8.2 ======= Installing dynamic plugin oci://fake-registry.example.com/my-org/my-plugin:1.2.3!some-plugin ==> Copying image oci://fake-registry.example.com/my-org/my-plugin:1.2.3 to local filesystem Traceback (most recent call last): File "/opt/app-root/src/install-dynamic-plugins.py", line 668, in install plugin_path = self.downloader.download(package) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/src/install-dynamic-plugins.py", line 604, in download tar_file = self.get_plugin_tar(image) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/src/install-dynamic-plugins.py", line 569, in get_plugin_tar self.skopeo(['copy', image_url, f'dir:{local_dir}']) File "/opt/app-root/src/install-dynamic-plugins.py", line 556, in skopeo rv = subprocess.run([self._skopeo] + command, check=True, capture_output=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/bin/skopeo', 'copy', 'docker://fake-registry.example.com/my-org/my-plugin:1.2.3', 'dir:/tmp/tmpd9_r7vz5/57fbb8e5c0cb641dcb19e63c89b16451fb013720ebdb92ed4ed333996e75f253']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/app-root/src/install-dynamic-plugins.py", line 1152, in <module> main() File "/opt/app-root/src/install-dynamic-plugins.py", line 1137, in main _, plugin_config = install_plugin(plugin, plugin_path_by_hash, dynamic_plugins_root, skip_integrity_check) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/src/install-dynamic-plugins.py", line 808, in install_plugin plugin_path = installer.install(plugin, plugin_path_by_hash) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/src/install-dynamic-plugins.py", line 684, in install raise InstallException(f"Error while installing OCI plugin {package}: {e}") InstallException: Error while installing OCI plugin oci://fake-registry.example.com/my-org/my-plugin:1.2.3!some-plugin: Command '['/usr/bin/skopeo', 'copy', 'docker://fake-registry.example.com/my-org/my-plugin:1.2.3', 'dir:/tmp/tmpd9_r7vz5/57fbb8e5c0cb641dcb19e63c89b16451fb013720ebdb92ed4ed333996e75f253']' returned non-zero exit status 1. ======= Removed lock file: /dynamic-plugins-root/install-dynamic-plugins.lock
Expected results:
In addition to the 'skopeo copy' exit code, we should also display its output (both stdout and stderr) in the init container logs, as this will be really handy for effective troubleshooting and hopefully quicker resolution.
If I run the same '['/usr/bin/skopeo', 'copy', 'docker://fake-registry.example.com/my-org/my-plugin:1.2.3', 'dir:/tmp/tmpd9_r7vz5/57fbb8e5c0cb641dcb19e63c89b16451fb013720ebdb92ed4ed333996e75f253']' command manually, I get this in stderr (expected):
FATA[0000] initializing source docker://fake-registry.example.com/my-org/my-plugin:1.2.3: pinging container registry fake-registry.example.com: Get "https://fake-registry.example.com/v2/": dial tcp: lookup fake-registry.example.com: no such host
We should see these lines in the install-dynamic-plugins init container logs as well.
Reproducibility (Always/Intermittent/Only Once):
Always
- relates to
-
-
- New
-
- links to
