Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-3523

Missing Tempo operator images when mirroring images using oc mirror, oc adm catalog mirror

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done-Errata
    • Undefined
    • rhosdt-3.0
    • rhosdt-2.9
    • Tempo
    • None
    • 2
    • False
    • None
    • False
    • Tracing Sprint # 243, Tracing Sprint # 244, Tracing Sprint # 245

    Description

      Description of the problem:

      When mirroring Tempo operator images to a mirror registry for use in a disconnected cluster, the related operator images for tempo, tempo-gateway, opa-openshift, tempo-query are not mirrored. 

      Version of components:

      OCP Server Version: 4.13.8

      OCP Client Version: 4.13.9

      Tempo operator version 0.3.0

      OPM version : version.Version{OpmVersion:"f1d9c59de", GitCommit:"f1d9c59de5a7cb32eb70846411cdc443e2b8f4b9", BuildDate:"2023-07-24T22:35:54Z", GoOs:"linux", GoArch:"amd64"}

      Steps to reproduce the issue:

      *Make sure you have a mirror registry or you can use your local registry for testing.

      *Create a directory to store the pruned index image configs.

      mkdir -p pruned-catalog/configs

      *Prune the stage registry registry.stage.redhat.io/redhat/redhat-operator-index:v4.13 to only select the desired operator packages.

      opm render registry.stage.redhat.io/redhat/redhat-operator-index:v4.13 | jq 'select(.package == "servicemeshoperator" or .package == "kiali-ossm" or .package == "tempo-product" or .package == "opentelemetry-product" or .package == "amq-streams" or .package == "jaeger-product" or .package == "elasticsearch-operator" or .name == "servicemeshoperator" or .name == "kiali-ossm" or .name == "tempo-product" or .name == "opentelemetry-product" or .name == "amq-streams" or .name == "jaeger-product" or .name == "elasticsearch-operator")' > pruned-catalog/configs/index.json

      *Generate the Dockerfile using the pruned index config.

      opm generate dockerfile pruned-catalog/configs

      *Build and publish the index image from the pruned-catalog directory.

      cd pruned-catalog/
      podman build -t quay.io/rhn_support_ikanse/test-catalog:latest -f configs.Dockerfile
      

      *Now mirror the images using the published index image to the mirror registry. Make sure you have the auth added in ~/.docker/config for connecting to the registry.

      oc adm catalog mirror quay.io/rhn_support_ikanse/test-catalog:latest bastion.iak413-upi.maistra.upshift.redhat.com:55555 --insecure=true --index-filter-by-os=linux/amd64

      *Check that there are no Tempo related images in the planning for images mirroring.

      phase 0:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch-operator-bundle   blobs=32  mounts=0 manifests=16  shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhn_support_ikanse/test-catalog                   blobs=16  mounts=0 manifests=1   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-operator-bundle                     blobs=10  mounts=0 manifests=5   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/opentelemetry-operator-bundle              blobs=10  mounts=0 manifests=5   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/tempo-operator-bundle                      blobs=4   mounts=0 manifests=3   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/3scale-istio-adapter-rhel8 blobs=8   mounts=0 manifests=6   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-operator-bundle        blobs=2   mounts=0 manifests=1   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-operator-bundle                  blobs=34  mounts=0 manifests=17  shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/strimzi-operator-bundle               blobs=8   mounts=0 manifests=4   shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/proxy-init-rhel7           blobs=116 mounts=0 manifests=61  shared=0
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/kiali-rhel8                blobs=324 mounts=0 manifests=180 shared=84
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch6-rhel8            blobs=144 mounts=0 manifests=84  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/kiali-rhel8-operator       blobs=300 mounts=0 manifests=90  shared=30
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-query-rhel8            blobs=12  mounts=0 manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-30-rhel8                   blobs=12  mounts=0 manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-istio/kiali-operator-metadata           blobs=37  mounts=0 manifests=18  shared=1
      phase 1:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/grafana-rhel8                 blobs=320 mounts=48 manifests=206 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/proxyv2-rhel8                 blobs=320 mounts=45 manifests=206 shared=51
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift4/ose-kube-rbac-proxy                       blobs=280 mounts=24 manifests=102 shared=34
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch-proxy-rhel8          blobs=160 mounts=32 manifests=96  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/kibana6-rhel8                      blobs=144 mounts=32 manifests=84  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/elasticsearch-rhel8-operator       blobs=160 mounts=32 manifests=96  shared=32
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-logging/logging-curator5-rhel8             blobs=240 mounts=28 manifests=84  shared=28
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/kiali-rhel7                   blobs=60  mounts=24 manifests=25  shared=24
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-ingester-rhel8                         blobs=42  mounts=15 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 ubi8/openjdk-11                                      blobs=32  mounts=12 manifests=12  shared=12
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-33-rhel8                      blobs=39  mounts=6  manifests=25  shared=9
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-es-rollover-rhel8         blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-rhel8-operator            blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-all-in-one-rhel8          blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-agent-rhel8               blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-es-index-cleaner-rhel8    blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-collector-rhel8           blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 distributed-tracing/jaeger-ingester-rhel8            blobs=12  mounts=6  manifests=5   shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/kafka-33-rhel8                           blobs=21  mounts=3  manifests=15  shared=3
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/strimzi-rhel8-operator                   blobs=21  mounts=3  manifests=15  shared=3
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/tempo-rhel8-operator                          blobs=9   mounts=3  manifests=10  shared=3
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/istio-rhel8-operator-metadata blobs=35  mounts=1  manifests=17  shared=1
      phase 2:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/prometheus-rhel8       blobs=302 mounts=76 manifests=191 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/istio-cni-rhel8        blobs=326 mounts=76 manifests=206 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/pilot-rhel8            blobs=320 mounts=76 manifests=206 shared=76
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/istio-rhel8-operator   blobs=174 mounts=72 manifests=85  shared=72
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/ratelimit-rhel8        blobs=246 mounts=66 manifests=150 shared=66
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/mixer-rhel8            blobs=104 mounts=40 manifests=56  shared=40
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-bridge-rhel8                 blobs=135 mounts=27 manifests=85  shared=33
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift4/ose-prometheus-config-reloader     blobs=56  mounts=22 manifests=18  shared=27
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/citadel-rhel8          blobs=44  mounts=22 manifests=21  shared=22
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/sidecar-injector-rhel8 blobs=44  mounts=22 manifests=21  shared=22
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/galley-rhel8           blobs=44  mounts=22 manifests=21  shared=22
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-rhel8-operator                  blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-all-in-one-rhel8                blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/opentelemetry-collector-rhel8          blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift-service-mesh/ior-rhel8              blobs=36  mounts=18 manifests=15  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-es-index-cleaner-rhel8          blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-agent-rhel8                     blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-query-rhel8                     blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-es-rollover-rhel8               blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/opentelemetry-rhel8-operator           blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 rhosdt/jaeger-collector-rhel8                 blobs=42  mounts=18 manifests=20  shared=18
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/bridge-rhel8                      blobs=24  mounts=6  manifests=15  shared=6
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq-streams/kafka-34-rhel8                    blobs=24  mounts=6  manifests=15  shared=6
      phase 3:
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-31-rhel8 blobs=111 mounts=33 manifests=65 shared=33
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-rhel8-operator blobs=141 mounts=33 manifests=90 shared=33
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 amq7/amq-streams-kafka-32-rhel8 blobs=129 mounts=27 manifests=85 shared=27
        bastion.iak413-upi.maistra.upshift.redhat.com:55555 openshift4/ose-prometheus       blobs=44  mounts=20 manifests=18 shared=20
      info: Planning completed in 7m25.29s
      

      *You can also check the rendered config from the stage index. Search for tempo-product and check the operator version related images.

      opm render registry.stage.redhat.io/redhat/redhat-operator-index:v4.13 > render.txt
          "relatedImages": [
              {
                  "name": "kube-rbac-proxy",
                  "image": "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:43dc8a63868bfb4b68901fa3edb492b966da4b3093e049eb595205dd9f4f1c7c"
              },
              {
                  "name": "tempo-rhel8-operator-17a5049192a56893d12be71a5556b152aea92eaaddaa6b33270a76e9a420d506-annotation",
                  "image": "registry.redhat.io/rhosdt/tempo-rhel8-operator@sha256:17a5049192a56893d12be71a5556b152aea92eaaddaa6b33270a76e9a420d506"
              },
              {
                  "name": "manager",
                  "image": "registry.redhat.io/rhosdt/tempo-rhel8-operator@sha256:17a5049192a56893d12be71a5556b152aea92eaaddaa6b33270a76e9a420d506"
              },
              {
                  "name": "",
                  "image": "registry.stage.redhat.io/rhosdt/tempo-operator-bundle@sha256:deebbfbc9ea63ec0f68dd899d91e07ad11ea2e10d29178cd0117c5b088e136db"
              }
          ]
      }
       

      Additional notes:

      To workaround this issue we need to specifically provide the Tempo operator related images in the mirroring tool so that the images can be mirrored to the registry. 

      Attachments

        Issue Links

          Activity

            People

              agerstma@redhat.com Andreas Gerstmayr
              rhn-support-ikanse Ishwar Kanse
              Ishwar Kanse Ishwar Kanse
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: