Description of problem:

Currently we don't have much information for oc-mirror metadata management best practices and pitfalls. We should enhance this part as this is always the error-prone area that caused many customer cases arisen. Also there are some changes introduced when downloading the content from official catalogs. We also need to describe some scenarios about different scenarios and pros and cons for each.

Version-Release number of selected component (if applicable):

4.11.latest

Additional info:

Below content should be considered to enhance our current documentation

1. Collocating the existing repos (oc-mirror unmanaged) with new repos (oc-mirror managed) in the same registry. According to jpower_personal , it is encouraged using oc-mirror a custom top-level namespace, especially if the registry has multiple users/teams who want to do their own management. Another reason to keep managed and unmanaged repos in a separate namespace is that oc-mirror won't be aware of unmanaged repos when pruning. If a child manifest was shared between a managed and unmanaged repo, the child manifest would be deleted during pruning if the managed repo was pruned. This will potentially cause the oc-mirror pruning operation accidentally touch and prune the unmanaged repos. Our documentation does not show an example of using custom top-level namespace by default, which is very likely followed by customers causing oc-mirror managed images and unmanaged images messed up together.
2. Mirroring strategy when there are a lot of contents (OCP release images, operator images, standalone images, helm charts) to be mirrored. This is a common scenario that a customer is planning to upgrade OCP along with their stacked operators. What is the best way to balance between mirroring efficiency and maintenance efficiency. I have concluded 3 possible scenarios. I will share details in later section.
3. Emphasize that no manual management of repos should be performed in an oc-mirror managed namespaces. This practice is discouraged because it will cause issues. I think this aligns the statement of When using the oc-mirror CLI plug-in to populate a mirror registry, any further updates to the mirror registry must be made using the oc-mirror tool.
4. Provide some mitigation steps in case customer already done some manual management (e.g customer deleted some managed repos from the registry, pushed a new tag). Those will cause inconsistency in the metadata. In such case, what is the best practice to keep the metadata back to consistency?
5. Although we do have described the two flags  "ignore-history" and "skip-metadata-check", normally customer does not know what the scenario is to use these two flags. For example, when customer face issue of "error: error during publishing, expecting imageset with prefix mirror_seq1: invalid mirror sequence order, want 1, got 2 " It is best to improve our error messages to prompt the users to add the flags on their own which is better than requesting help from Red Hat Support. That will help customer self-sufficiency.  But anyway I understand this is more like a code enhancement, not from documentation part
6. [Important] Describe the full metadata management lifecycle. For example, when the oc-mirror first populates images to the registry, it will create the oc-mirror repo and populate a metadata image with a UUID tag, then when the next push is being performed, the metadata on the registry will be compared with the metadata included in the image archive tar file to determine whether incremental images mirroring or pruning is required.  According to GitHub the location of metadata on the registry is located on <user-defined registry>/<user-defined namespace>/oc-mirror:<metadata-uuid>. The uuid can be obtained using the command "oc-mirror describe"
7. We may also need to explain how the metadata image is determined to perform the operations.
8. Since the metadata is the foundation of incremental mirroring and pruning,  we need to emphasize the importance of maintaining this repository on the registry. Highlight that this repository includes the current managed images manifests by oc-mirror. Deleting this repository will cause the information losing and all images uploaded previously will become unmanaged causing the future uploading of images unable to do manifests comparison, eventually causing the incremental mirroring and pruning functions unable to work correctly. 

For the Mirroring strategy described in #2:

Scenario 1:  write a full imageset yaml file

Put all mirroring information in one file (OCP release images, Operator images ,etc). In case there is need to download new OCP release or operator images, make it as the single source of truth. This will put all images into one archive tar file.

Scenario 2:  Incremental way to add more configuration gradually.

Write a small imageset yaml file first which include a piece of images (for example, if customer is going to upgrade to OCP 4.10 from OCP 4.8, first mirror OCP 4.9 release images). Run the mirror-to-disk to generate a 001 archive tar file and then disk-to-mirror, when mirroring, specify a custom top-level namespace to isolate. But do not need to run the ICSP and Catalog Source yaml file at this time. Then, back to the imageset yaml to include additional components and repeat above steps. Specify a different custom top-level namespace. Finally manually merge the ICSP and Catalog Source yaml files and execute only once.  In this approach, all the images share the same metadata file.

In this approach, I'm told that the order is important - we must first push the 001 archive tar file then 002 archive. If the order is not maintained , there might be issues of "invalid mirroring sequence" error.

Scenario 3: Using different working folder

Similar as Scenario 2, but we split different operators/release images into different working folder. In this approach, metadata file is NOT shared. Also use different custom top-level namespace when pushing.

Summarizing pros and cons for above 3 scenarios:

To summary, for the metadata management, we need to include below information:

• What is metadata in oc-mirror and what information is included
• Why metadata is important
• How the metadata plays the role during image operation process
• Best practice to maintain metadata
  • no manual maintenance
  • any necessity of backup & restore metadata on the registry 
  • moving metadata between different working folders (local backup)
  • any other..
• Consider different customer scenarios above, what is the best approach to balance image pushing efficiency and local metadata maintenance efficiency
• Some common troubleshooting steps in case of metadata corruption & lost & inconsistency
  • For full disconnected env, we can use "oc-mirror describe <archive file>" to view its source
  • Is there any way to view and export the metadata information on the oc-mirror repo in registry?
  • The applicable scenarios of using "ignore-history" and "skip-metadata-check" with detailed explanation (also side effect of using these flags) 
• during dry-run, can we summarize how many images are to-be-pushed and to-be-pruned (more like a new RFE..)