Original reporter: davidkirwanirl@gmail.com

The issue:

Every night we do a backup. This involves a cronjob running on the os-control or os-control.stg systems which execs inside the forgejo container and runs `forgejo dump`, which creates a DB dump along with all the repos, then zips it all up. The cron then rsyncs it out of the container to the os-control system.

A second cronjob prunes and manages the backups. It does the following:

• keep last 7 daily backups
• every week, take the last daily and make it a weekly, then prune the rest
• keep the last 4 weekly backups and once a month make it a monthly, then prune the other weekly backups
• keep the last 12 monthly backups, once a year make the last monthly a yearly backup, prune the other monthly backups
• keep all the yearly backups indefinitely.

My original idea was to then keep all these in AWS S3, but having an issue with the pruning aspect, as I can't delete from the mounted S3 filesystem.. so I need to revisit that and figure out what to do about it.

These backups in prod are already 25GB and rising as more people use the forge. This might take a while to stabilise but we are already using more space than available on the os-control system. Hoping that the solution will also then also work for the distgit instances coming down the line.