For OCP-72942 , updated
[knarra] Thanks !!

For OCP-72615 :
[knarra] Got it, thanks !!

Hope moving the directory to another dir and running the command again is how you test the cache.

For OCP-72942 , updated ;

For OCP-72615 :

1) we have other case cover the `release image ` test ;

2) we have cover the multiple catalog in case OCP-72971 

3)  this case if the full work flow test for v2 , prepare command is to  query images from local cache . 

4)  we can specify the port not use the default 55000;

https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-72942
[comments]

• can we also try to test with a number greater than 2 ? you could refer to the bug here which customer was facing https://issues.redhat.com//browse/OCPBUGS-11922
• Also could you change imageSet to include `local-storage-operator` from the above bug i see customer faced issue when using `local-storage-operator` with -max-nested-path as 3
• should we also include release images as well into the imageSetConfig ?

https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-72615
[comments]

• change the title of the test case to include `release images as well`
• you must include more than one operator catalog in the imageSetConfig file to be able to verify if catalogSource is generated for each catalog
• Testcase name does not talk anything about prepare and what it does, better to include this as a separate case and also include what happens if certain operator/image is not present in the local cache. Also explain what is being done with `--prepare` as it is not clear from the expected results.
• I think port should be `55000` and not 5005 ?

Verify `no resolution of tags into digest` - might need a test for this, could check with dev on how we could verify this
Did not see a testcase to test for `targetCatalogSourceTemplate` could you please help check ? thanks !!

knarra@redhat.com   test cases : 

https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitems/testcase?query=trello%3ACLID%5C-10

yinzhou@redhat.com Any tests for this epic here ? could you please help link them here ?

I do see changes like adding a new field `targetCatalogSourceTemplate` and not rebuilding catalogs (may be we need to verify this so that we can be sure of it, if there is a way to do so), one catalog resource per catalog, no resolution of tags into digest

Demo link :

https://drive.google.com/file/d/1nIhZaZxzvDd8MEyX5wetvj33eEawaRYV/view?usp=drive_link

rhn-support-stk rhn-support-snarayan 
For more info, please take a look at the comments, and also at the descriptions in the PR. The latter have examples of inputs and expected outputs.

PS: please take into account this https://github.com/openshift/oc-mirror/pull/784#issuecomment-1924346817

Adding an updateStrategy to catalogSource files generated by oc-mirror

Hi qiwan233
Can you help us validate that this is what OCP is expecting from the mirroring done by oc-mirror?

Here are the questions we are having:

1. We heard from somewhere that registries can have images with no tag nor digest. Can this happen? can you explain in which case? and what such an image is?
2. Similar: can a registry have an image with tag only? no digest?
3. CFE-817 (the original study) established that by default, oc-mirror will generate IDMS only. Images referenced by tag will be added to that IDMS. We are questioning this choice now:
   1. Do you agree that if a cluster deploys such an IDMS, and references those images by tag, the mirror will not be used? qiwan233 : Yes
   2. If above is true, do you advise that the default behavior of oc-mirror should be to
      1. keep the behavior
      2. generate IDMS and ITMS (images by tag can therefore benefit from the mirrors)
      3. generate IDMS only, exclude images by tag and display warnings for them
   3. rhn-engineering-mitr :  (It’s trivial to define ITMS manually, so I don’t think this is worth worrying that much about.) Sure, having oc mirror generate an appropriate ITMS (or both — either just output all of them and let the user filter, or provide a ≤5-way CLI option to choose between no output / ICSP / IDMS / ITMS / IDMS+ITMS) would be convenient to some users. Release payload all uses digest references, to ITMS will, at worst, not work; having ITMS defined should not break anything, I think?
   4. Decision: Always generate IDMS and ITMS. The latter will be generated if images by tag are encountered
4. In the legacy oc-mirror, oc-mirror was resolving an image tag to digest prior to mirroring, AND rebuilding operator catalogs (modifies all references to images to references by digest). I think this explains why they could add all images in ICSP
   1. We are thinking about stopping this in the new implementation: tags will be preserved on the destination mirror. Do you agree?
   2. rhn-engineering-mitr : No opinion. I do think it is valuable to preserve original images (and keep their signatures working, for supply chain security auditability), OTOH I don’t know if there are any users who might rely on the digest resolution feature. Maybe the mirroring and the digest resolution should be maintained and/or dropped as separate features?
   3. Decision:No resolution of tags into digests. Images remain the way they are referenced. Especially that we generate both ITMS and IDMS{}
5. When an image is referenced only by digest, it seems that it is not visible in some registries. oc was adding a fake tag (truncated digest) for all images except release images so that images can be visible on registries. Do you think this practice is good? should we keep it in the new design?
   1. The reason we are asking this is because as you can conclude from questions 4 and 5, legacy oc-mirror ends up removing a tag and adding another...
   2. rhn-engineering-mitr :  It’s not just visibility. Some registries, AFAIK (and notably including Quay.io), semi-regularly remove manifests with no tag attached (assuming that they are unreachable and/or a part of an aborted upload). So I think the feature to add a tag is valuable and probably needs to be preserved. (It might not need to be enabled by default nor hard-coded to on, but then existence of those tags shouldn’t hurt anything.)

Thank you for your answers

Sherine

IDMS / ITMS Generation

From study in CFE-817, the design is adapted for V2 as follows:

• --skip-image-pin is not relevant for v2
• all images by tag are added to ITMS, while all images by digest are added to IDMS
• By default, ImageDigestMirrorSet is generated, and contains mirrors for images referenced by digest in the release, operator catalog or additional images, helm charts
• By default, ImageTagMirrorSet is generated if at least 1 image referenced by tag was found. ITMS contains mirrors for images referenced by tag in the release, operator catalog or additional images, helm charts
• IDMS and ITMS use source and mirror paths scoped to namespace , unless --max-nested-paths has been used. In the latter case, repository scope is used.
• V2 doesn't do tag to digest resolution prior to mirroring. In V1, oc-mirror was replacing tags by digests before handing mappings.txt to the oc code, which in turn, created a fake tag (derived from the digest value). This V1 behavior is not considered fit for V2 because:
  • we have ITMS,
  • we don't call oc,
  • we don't rebuild catalogs so far (the reference for the image is not modified in the declarative config).
• Naming convention:
  • oc-mirror v2 will generate 1 itms-oc-mirror.yaml + 1 idms-oc-mirror.yaml
  • each file will contain several custom resources, 1 per category of images mirrored
    • itms-generic-0 / idms-generic-0
    • itms-operator-0 / idms-operator-0
    • itms-release-0 / idms-release-0
  • oc-mirror v2 will not chunk IDMS/ITMS resources by size like in v1, where an arbitrary chunking at 250k was performed: these manifests will rarely reach this size.

See diagram for more details

Catalog Source Generation

In V2, it is possible to add a targetCatalogSourceTemplate for each catalog in the imageSetConfig, in order to allow the user customization of the catalogSource (setting updateStrategy, or grpcPodConfig)

• • if not provided, the catalogSource will just contain the reference to the image, alongside the sourceType (grpc)
  • when provided, oc-mirror will attempt to use the template as a base for catalogSource generation, and in case of failure, it will failover to generation without template with a warning in the logs

Example ImageSetConfig:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror: 
  operators: 
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
    packages: 
    - name: aws-load-balancer-operator
    targetCatalogSourceTemplate: "/home/skhoury/go/src/github.com/openshift/oc-mirror/v2/tests/catalog-source_template.yaml"

Example template

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata: 
  name: totalRubbish
  namespace: openshift-marketplace
spec: 
  image: totalRubbish
  sourceType: grpc
  updateStrategy: 
    registryPoll: 
      interval: 30m0s

Naming convention for catalogSource resources:

• oc-mirror v2 generates 1 catalogSource file (and custom resource) per catalog
• the name of the custom resource and file is generated as a concatenation of:
  • "cs-"
  • The name of the catalog repository. Ex: for registry.redhat.io/redhat/redhat-operator-index:v4.14, "redhat-operator-index" will be used
  • "-" to separate the catalog repo from the tag part
  • The tag of the catalog repository, after replacing "." by "-". Example for registry.redhat.io/redhat/redhat-operator-index:v4.14, "v4-14" will be used