DOCUMENT backup/restore requirements for Quay

There are several secnarios that need to be covered:
a) Quay is completely deployed on VMs
b) Quay is deployed on OpenShift using the operator:
b1) the operator does not manage the db.
b2) the operator manages the db.

There are several things that need to be backed up:
a) for VM deployments: config bundle directory (directory that bind mounts to /conf/stack), full DB backup, full blob storage backup (if public cloud storage is used, this may not be needed since Azure, AWS and Google have a bunch of policies that disallow removal of buckets).
b) for OpenShift deployment: custom config bundle (config options that we send to the operator), QuayRegistry CR, full database backup, full blob storage backup (again, for public cloud storage this may not be needed).

Restore procedure (we will assume everything's lost):
a) VM deployment:

• create a new database in the database engine and create the pg_trgm extension on that database
• restore the Quay database from backup with psql
• create a new blob storage bucket and copy all blobs to the bucket from backup with the appropriate tool (such as s3cmd, awscli, azure)
• restore the config bundle directory from storage. Edit the DB_URI inside the config.yaml file so it references the new database instance (if needed). Edit the storage parameters (new bucket name, new host, new credentials) if needed.
• start the same version of Quay as it was run before the incident against the restored config bundle
• if Quay restarts normally and UI logon and push/pull works, rescale Quay back to the original number.

b) Openshift deployment:
b1) when the operator does not manage the db:

• create a new database in the database engine and create the pg_trgm extension on that database
• restore the Quay database from backup with psql
• create a new blob storage bucket and copy all blobs to the bucket from backup with the appropriate tool (such as s3cmd, awscli, azure)
• edit the custom config bundle and modify DB_URI and storage parameters if needed. Create the custom config bundle secret with the same name as before.
• install the Quay operator and apply the same QuayRegistry CR as before (it should reference the created custom config bundle). The operator should reconcile everything and create necessary secrets and start Quay.

b2) when the operator manages the db:

• install the operator and apply the default QuayRegistry CR. The operator will deploy all components and start Quay.
• once Quay is started, downscale the operator deployment, Quay main app deployment and Quay mirror deployment.
• copy the database backup to /var/lib/pgsql/data/userdata directory inside the deployed PostgreSQL pod:

oc get pods
oc cp /path/to/Quay/db/backup.sql REGISTRY_NAME-quay-database-xxxxxxxx-xxxx:/var/lib/pgsql/data/userdata/

• execute a shell in the PostgreSQL pod and start psql. Dump the automatically created database, recreate it with the same name and owner and create the pg_trgm extension on it.
• Once done, exit psql and import the backup into PostgreSQL. Exit the pod shell.
• Create the custom config bundle you retrieved from backup. Edit the deployed QuayRegistry CR and add the bundle reference to it. Rescale the operator back to 1 pod.
• The operator should start, it should reconcile and create new secrets and Quay should start after that with the restored db.

These procedures should be tested before they become part of the official docs.