Loading...

Type: Bug
Resolution: Not a Bug
Priority: Major
Fix Version/s: None
Affects Version/s: quay-v3.5.7, quay-v3.6.0
Component/s: quay
Labels:
- quay-enterprise
- triaged

Blocked:
False
Ready:
False
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

After enabling the Action Log Rotation and Archiving function, the logs about push image operations have not been archived in the storage. The archive directory has not even been created in storage.

Version-Release number of selected component (if applicable):

Quay 3.6.0
Quay 3.5.7

How reproducible:

always

Steps to Reproduce:
1. Deploy quay 3.6.0 in openshift by operatorhub
2. After quay work, created an account "whuquay" and push a image to the repository under the "whuquay" organization.
3. Enable Action Log Rotation and Archiving function in quay config editor UI

  Checked "Enable Action Log Rotation" option
  Storage location: default
  Storage path: archives/actionlogs
  Log Rotation Threshold: 1m

4. Click "Validate Configuration Changes" button, after passing the validation, click Reconfigure Quay button.
5. After operator reconciled all pods, waiting for more than 1 minute
6. Try to find archive logs in storage

Actual results:

There was no "archives/actionlogs" directory in storage.

The screenshot of AWS S3 bucket

Expected results:

There should be a "archives/actionlogs" directory in the storage and the push operation log should be saved in this directory

Additional info:

$ oc get secret $(oc get quayregistry -n quay-enterprise -o jsonpath='{.items[0].spec.configBundleSecret}') -n quay-enterprise -o jsonpath='{.data.config\.yaml}' | base64 -d
ACTION_LOG_ARCHIVE_LOCATION: default
ACTION_LOG_ARCHIVE_PATH: archives/actionlogs
ACTION_LOG_ROTATION_THRESHOLD: 1m
ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
AUTHENTICATION_TYPE: Database
AVATAR_KIND: local
DATABASE_SECRET_KEY: YPQZ0hFUM1.....KPyC3xw-GRjmX7ixAKRS9x
DEFAULT_TAG_EXPIRATION: 2w
DISTRIBUTED_STORAGE_CONFIG:
 default:
 - S3Storage
 - host: s3.us-east-2.amazonaws.com
 s3_access_key: AK...UP
 s3_bucket: whuaws
 s3_secret_key: cQ....CV4
 storage_path: /quaydata
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- default
DISTRIBUTED_STORAGE_PREFERENCE:
- default
ENTERPRISE_LOGO_URL: /static/img/quay-horizontal-color.svg
EXTERNAL_TLS_TERMINATION: true
FEATURE_ACTION_LOG_ROTATION: true
FEATURE_ANONYMOUS_ACCESS: true
FEATURE_APP_SPECIFIC_TOKENS: true
FEATURE_BITBUCKET_BUILD: false
FEATURE_BLACKLISTED_EMAILS: false
FEATURE_BUILD_SUPPORT: false
FEATURE_CHANGE_TAG_EXPIRATION: true
FEATURE_DIRECT_LOGIN: true
FEATURE_EXTENDED_REPOSITORY_NAMES: true
FEATURE_FIPS: false
FEATURE_GITHUB_BUILD: false
FEATURE_GITHUB_LOGIN: false
FEATURE_GITLAB_BUILD: false
FEATURE_GOOGLE_LOGIN: false
FEATURE_INVITE_ONLY_USER_CREATION: false
FEATURE_MAILING: false
FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
FEATURE_PARTIAL_USER_AUTOCOMPLETE: true
FEATURE_PROXY_STORAGE: false
FEATURE_SECURITY_NOTIFICATIONS: true
FEATURE_SIGNING: false
FEATURE_STORAGE_REPLICATION: false
FEATURE_TEAM_SYNCING: false
FEATURE_USER_CREATION: true
FEATURE_USER_LAST_ACCESSED: true
FEATURE_USER_LOG_ACCESS: false
FEATURE_USER_METADATA: false
FEATURE_USER_RENAME: false
FEATURE_USERNAME_CONFIRMATION: true
FRESH_LOGIN_TIMEOUT: 10m
GITHUB_LOGIN_CONFIG: {}
GITHUB_TRIGGER_CONFIG: {}
GITLAB_TRIGGER_KIND: {}
GPG2_PRIVATE_KEY_FILENAME: signing-private.gpg
GPG2_PUBLIC_KEY_FILENAME: signing-public.gpg
LDAP_ALLOW_INSECURE_FALLBACK: false
LDAP_EMAIL_ATTR: mail
LDAP_UID_ATTR: uid
LDAP_URI: ldap://localhost
LOGS_MODEL: database
LOGS_MODEL_CONFIG: {}
MAIL_DEFAULT_SENDER: support@quay.io
MAIL_PORT: 587
MAIL_USE_AUTH: false
MAIL_USE_TLS: false
PREFERRED_URL_SCHEME: https
REGISTRY_TITLE: Quay
REGISTRY_TITLE_SHORT: Quay
SEARCH_MAX_RESULT_PAGE_COUNT: 10
SEARCH_RESULTS_PER_PAGE: 10
SECRET_KEY: -cq03J...JE
SECURITY_SCANNER_V4_PSK: UG....Q=
SERVER_HOSTNAME: quayregistry-quay-quay-enterprise.apps.whu48aws22.qe.devcluster.openshift.com
SETUP_COMPLETE: true
SUPER_USERS:
- whuquay
TAG_EXPIRATION_OPTIONS:
- 2w
TEAM_RESYNC_STALE_TIME: 60m
TESTING: false
USER_RECOVERY_TOKEN_LIFETIME: 30m

Just found some logs about buildlogsarchiver and exportactionlogsworker in quay pod.

buildlogsarchiver stdout | 2021-10-28 02:22:50,257 [59] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2021-10-28 02:23:20 UTC)" (scheduled at 2021-10-28 02:22:50.256825+00:00)
buildlogsarchiver stdout | 2021-10-28 02:22:50,259 [59] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2021, 10, 13, 2, 22, 50, 258071), False, 50, 1, 0])
buildlogsarchiver stdout | 2021-10-28 02:22:50,266 [59] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2021-10-28 02:22:50,266 [59] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2021-10-28 02:22:50,266 [59] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2021-10-28 02:23:20 UTC)" executed successfully
buildlogsarchiver stdout | 2021-10-28 02:23:11,177 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-d65d45cbb-b8cvn', 'process_name': 'buildlogsarchiver.py', 'pid': '59'}
buildlogsarchiver stdout | 2021-10-28 02:23:20,257 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2021-10-28 02:23:20,257 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2021-10-28 02:23:50.256825+00:00 (in 29.999404 seconds)
exportactionlogsworker stdout | 2021-10-28 02:22:23,551 [63] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2021-10-28 02:23:23 UTC)" (scheduled at 2021-10-28 02:22:23.550214+00:00)
exportactionlogsworker stdout | 2021-10-28 02:22:23,551 [63] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2021-10-28 02:22:23,552 [63] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2021, 10, 28, 2, 22, 23, 551375), True, datetime.datetime(2021, 10, 28, 2, 22, 23, 551375), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2021-10-28 02:22:23,559 [63] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2021-10-28 02:22:23,559 [63] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2021-10-28 02:22:23,559 [63] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2021-10-28 02:23:23 UTC)" executed successfully
exportactionlogsworker stdout | 2021-10-28 02:22:32,164 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-d65d45cbb-b8cvn', 'process_name': 'exportactionlogsworker.py', 'pid': '63'}
exportactionlogsworker stdout | 2021-10-28 02:22:54,142 [63] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2021-10-28 02:22:54,142 [63] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2021-10-28 02:23:23.550214+00:00 (in 29.407521 seconds)