Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-3523

Missing Tempo operator images when mirroring images using oc mirror, oc adm catalog mirror

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • rhosdt-3.0
    • rhosdt-2.9
    • Tempo
    • None
    • 2
    • False
    • None
    • False
    • Tracing Sprint # 243, Tracing Sprint # 244, Tracing Sprint # 245

      Description of the problem:

      When mirroring Tempo operator images to a mirror registry for use in a disconnected cluster, the related operator images for tempo, tempo-gateway, opa-openshift, tempo-query are not mirrored. 

      Version of components:

      OCP Server Version: 4.13.8

      OCP Client Version: 4.13.9

      Tempo operator version 0.3.0

      OPM version : version.Version{OpmVersion:"f1d9c59de", GitCommit:"f1d9c59de5a7cb32eb70846411cdc443e2b8f4b9", BuildDate:"2023-07-24T22:35:54Z", GoOs:"linux", GoArch:"amd64"}

      Steps to reproduce the issue:

      *Make sure you have a mirror registry or you can use your local registry for testing.

      *Create a directory to store the pruned index image configs.

      mkdir -p pruned-catalog/configs

      *Prune the stage registry registry.stage.redhat.io/redhat/redhat-operator-index:v4.13 to only select the desired operator packages.

      opm render registry.stage.redhat.io/redhat/redhat-operator-index:v4.13 | jq 'select(.package == "servicemeshoperator" or .package == "kiali-ossm" or .package == "tempo-product" or .package == "opentelemetry-product" or .package == "amq-streams" or .package == "jaeger-product" or .package == "elasticsearch-operator" or .name == "servicemeshoperator" or .name == "kiali-ossm" or .name == "tempo-product" or .name == "opentelemetry-product" or .name == "amq-streams" or .name == "jaeger-product" or .name == "elasticsearch-operator")' > pruned-catalog/configs/index.json

      *Generate the Dockerfile using the pruned index config.

      opm generate dockerfile pruned-catalog/configs

      *Build and publish the index image from the pruned-catalog directory.

      cd pruned-catalog/
      podman build -t quay.io/rhn_support_ikanse/test-catalog:latest -f configs.Dockerfile
      

      *Now mirror the images using the published index image to the mirror registry. Make sure you have the auth added in ~/.docker/config for connecting to the registry.

      oc adm catalog mirror quay.io/rhn_support_ikanse/test-catalog:latest bastion.iak413-upi.maistra.upshift.redhat.com:55555 --insecure=true --index-filter-by-os=linux/amd64

      *Check that there are no Tempo related images in the planning for images mirroring.

      phase 0:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch-operator-bundle   blobs=32  mounts=0 manifests=16  shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhn_support_ikanse/test-catalog                   blobs=16  mounts=0 manifests=1   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-operator-bundle                     blobs=10  mounts=0 manifests=5   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/opentelemetry-operator-bundle              blobs=10  mounts=0 manifests=5   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/tempo-operator-bundle                      blobs=4   mounts=0 manifests=3   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/3scale-istio-adapter-rhel8 blobs=8   mounts=0 manifests=6   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-operator-bundle        blobs=2   mounts=0 manifests=1   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-operator-bundle                  blobs=34  mounts=0 manifests=17  shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/strimzi-operator-bundle               blobs=8   mounts=0 manifests=4   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/proxy-init-rhel7           blobs=116 mounts=0 manifests=61  shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/kiali-rhel8                blobs=324 mounts=0 manifests=180 shared=84
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch6-rhel8            blobs=144 mounts=0 manifests=84  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/kiali-rhel8-operator       blobs=300 mounts=0 manifests=90  shared=30
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-query-rhel8            blobs=12  mounts=0 manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-30-rhel8                   blobs=12  mounts=0 manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-istio/kiali-operator-metadata           blobs=37  mounts=0 manifests=18  shared=1
      phase 1:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/grafana-rhel8                 blobs=320 mounts=48 manifests=206 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/proxyv2-rhel8                 blobs=320 mounts=45 manifests=206 shared=51
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift4/ose-kube-rbac-proxy                       blobs=280 mounts=24 manifests=102 shared=34
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch-proxy-rhel8          blobs=160 mounts=32 manifests=96  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/kibana6-rhel8                      blobs=144 mounts=32 manifests=84  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch-rhel8-operator       blobs=160 mounts=32 manifests=96  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/logging-curator5-rhel8             blobs=240 mounts=28 manifests=84  shared=28
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/kiali-rhel7                   blobs=60  mounts=24 manifests=25  shared=24
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-ingester-rhel8                         blobs=42  mounts=15 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 ubi8/openjdk-11                                      blobs=32  mounts=12 manifests=12  shared=12
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-33-rhel8                      blobs=39  mounts=6  manifests=25  shared=9
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-es-rollover-rhel8         blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-rhel8-operator            blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-all-in-one-rhel8          blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-agent-rhel8               blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-es-index-cleaner-rhel8    blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-collector-rhel8           blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-ingester-rhel8            blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/kafka-33-rhel8                           blobs=21  mounts=3  manifests=15  shared=3
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/strimzi-rhel8-operator                   blobs=21  mounts=3  manifests=15  shared=3
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/tempo-rhel8-operator                          blobs=9   mounts=3  manifests=10  shared=3
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/istio-rhel8-operator-metadata blobs=35  mounts=1  manifests=17  shared=1
      phase 2:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/prometheus-rhel8       blobs=302 mounts=76 manifests=191 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/istio-cni-rhel8        blobs=326 mounts=76 manifests=206 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/pilot-rhel8            blobs=320 mounts=76 manifests=206 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/istio-rhel8-operator   blobs=174 mounts=72 manifests=85  shared=72
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/ratelimit-rhel8        blobs=246 mounts=66 manifests=150 shared=66
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/mixer-rhel8            blobs=104 mounts=40 manifests=56  shared=40
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-bridge-rhel8                 blobs=135 mounts=27 manifests=85  shared=33
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift4/ose-prometheus-config-reloader     blobs=56  mounts=22 manifests=18  shared=27
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/citadel-rhel8          blobs=44  mounts=22 manifests=21  shared=22
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/sidecar-injector-rhel8 blobs=44  mounts=22 manifests=21  shared=22
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/galley-rhel8           blobs=44  mounts=22 manifests=21  shared=22
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-rhel8-operator                  blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-all-in-one-rhel8                blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/opentelemetry-collector-rhel8          blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/ior-rhel8              blobs=36  mounts=18 manifests=15  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-es-index-cleaner-rhel8          blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-agent-rhel8                     blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-query-rhel8                     blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-es-rollover-rhel8               blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/opentelemetry-rhel8-operator           blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-collector-rhel8                 blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/bridge-rhel8                      blobs=24  mounts=6  manifests=15  shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/kafka-34-rhel8                    blobs=24  mounts=6  manifests=15  shared=6
      phase 3:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-31-rhel8 blobs=111 mounts=33 manifests=65 shared=33
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-rhel8-operator blobs=141 mounts=33 manifests=90 shared=33
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-32-rhel8 blobs=129 mounts=27 manifests=85 shared=27
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift4/ose-prometheus       blobs=44  mounts=20 manifests=18 shared=20
      info: Planning completed in 7m25.29s
      

      *You can also check the rendered config from the stage index. Search for tempo-product and check the operator version related images.

      opm render registry.stage.redhat.io/redhat/redhat-operator-index:v4.13 > render.txt
          "relatedImages": [
              {
                  "name": "kube-rbac-proxy",
                  "image": "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:43dc8a63868bfb4b68901fa3edb492b966da4b3093e049eb595205dd9f4f1c7c"
              },
              {
                  "name": "tempo-rhel8-operator-17a5049192a56893d12be71a5556b152aea92eaaddaa6b33270a76e9a420d506-annotation",
                  "image": "registry.redhat.io/rhosdt/tempo-rhel8-operator@sha256:17a5049192a56893d12be71a5556b152aea92eaaddaa6b33270a76e9a420d506"
              },
              {
                  "name": "manager",
                  "image": "registry.redhat.io/rhosdt/tempo-rhel8-operator@sha256:17a5049192a56893d12be71a5556b152aea92eaaddaa6b33270a76e9a420d506"
              },
              {
                  "name": "",
                  "image": "registry.stage.redhat.io/rhosdt/tempo-operator-bundle@sha256:deebbfbc9ea63ec0f68dd899d91e07ad11ea2e10d29178cd0117c5b088e136db"
              }
          ]
      }
       

      Additional notes:

      To workaround this issue we need to specifically provide the Tempo operator related images in the mirroring tool so that the images can be mirrored to the registry. 

              agerstma@redhat.com Andreas Gerstmayr
              rhn-support-ikanse Ishwar Kanse
              Ishwar Kanse Ishwar Kanse
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: