Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-5351

uploading docker file got 500 error page when use managed ODF storage

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • quay-v3.8.6
    • quay-builder
    • None
    • False
    • None
    • False

      Description of problem:

      When set up quay with managed ODF storage, enable build image function, upload docker file to trigger build process, will get 500 error page.

      Version-Release number of selected component (if applicable):

       

      //cvp-redhatquay: quay-operator-bundle-container-v3.8.6-4
      //Index image v4.13: registry-proxy.engineering.redhat.com/rh-osbs/iib:484381
      
      ------------------------------
      $ oc get $(oc get pod -l name=quay-operator-alm-owned -o name -n quay-enterprise) -o jsonpath='{.spec.containers[0].image}' -n quay-enterprise
      registry.redhat.io/quay/quay-operator-rhel8@sha256:b078a3d2809f82886d70e0f5e780214a747844420ce030e382dc2e76336e76bd
      
      ------------------------------
      $ oc get $(oc get pod -l app=quay -o NAME -n quay-enterprise|head -n 1) -o jsonpath='{.spec.containers[0].image}' -n quay-enterprise
      registry.redhat.io/quay/quay-rhel8@sha256:4ecfeaad7e8909b1a166fa0c9b3aa75550d353f1aac1971beaf79e06977c9e84
      
      ------------------------------
      $ oc get quayregistry -o jsonpath='{.items[0].status.currentVersion}'  -n quay-enterprise
      V3.8.6

       

      How reproducible:

      always

      Steps to Reproduce:
      1.  configure quay with managed ODF storage (noobaa) and enable build image function

       

      $ oc -n quay-enterprise rsh $(oc get pod -l app=quay -o NAME -n quay-enterprise|head -n 1) cat /conf/stack/config.yaml
      ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
      AUTHENTICATION_TYPE: Database
      BITBUCKET_TRIGGER_CONFIG:
        CONSUMER_KEY: H...j
        CONSUMER_SECRET: .....
      BUILD_MANAGER:
      - ephemeral
      - ALLOWED_WORKER_COUNT: 20
        EXECUTORS:
        - BUILDER_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.8.6
          BUILDER_NAMESPACE: virtual-builds
          CONTAINER_CPU_LIMITS: 1000m
          CONTAINER_CPU_REQUEST: 500m
          CONTAINER_MEMORY_LIMITS: 1G
          CONTAINER_MEMORY_REQUEST: 1G
          DEBUG: true
          EXECUTOR: kubernetesPodman
          K8S_API_SERVER: api.whu413gcp0.qe.gcp.devcluster.openshift.com:6443
          K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt
          KUBERNETES_DISTRIBUTION: openshift
          NAME: openshift
          NODE_SELECTOR_LABEL_KEY: ""
          NODE_SELECTOR_LABEL_VALUE: ""
          QUAY_PASSWORD: .......
          QUAY_USERNAME: .......
          SERVICE_ACCOUNT_NAME: quay-builder
          SERVICE_ACCOUNT_TOKEN: e......-k
          SETUP_TIME: 180
          VOLUME_SIZE: 8G
        ORCHESTRATOR:
          REDIS_HOST: quayregistry-quay-redis
          REDIS_PASSWORD: ""
          REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
          REDIS_SSL: false
        ORCHESTRATOR_PREFIX: buildman/production/
      BUILDLOGS_REDIS:
        host: quayregistry-quay-redis
        port: 6379
      BUILDMAN_HOSTNAME: quayregistry-quay-builder-quay-enterprise.apps.whu413gcp0.qe.gcp.devcluster.openshift.com:443
      DATABASE_SECRET_KEY: o02o6-OHVfvpuS1iTUjK29Bpe7Tb721Mv7NVciSVBH2auWFcDXVAq1gC3paotOgP5TZImQ5ugVh8hm7s
      DB_CONNECTION_ARGS:
        autorollback: true
        threadlocals: true
      DB_URI: postgresql://quayregistry-quay-database:QzgZKMyK3D5O6e0BPprS7T0MjrhyX0nmJuTNAZgleZ6wPGPQAiBvhGY2vZL-vcZyet84Gxc8-peDlg-H@quayregistry-quay-database:5432/quayregistry-quay-database
      DEFAULT_TAG_EXPIRATION: 2w
      DISTRIBUTED_STORAGE_CONFIG:
        local_us:
        - RHOCSStorage
        - access_key: GsCi7SrJMyVeczw2bgjS
          bucket_name: quay-datastore-76136d4a-c7e8-4b70-a095-fd8284616d9f
          hostname: s3.openshift-storage.svc.cluster.local
          is_secure: true
          port: 443
          secret_key: wVKsjGENg7cBhQTrekei953z92P3unZF12SLkorU
          storage_path: /datastorage/registry
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
      - local_us
      DISTRIBUTED_STORAGE_PREFERENCE:
      - local_us
      ENTERPRISE_LOGO_URL: /static/img/RH_Logo_Quay_Black_UX-horizontal.svg
      EXTERNAL_TLS_TERMINATION: false
      FEATURE_BITBUCKET_BUILD: true
      FEATURE_BUILD_SUPPORT: true
      FEATURE_DIRECT_LOGIN: true
      FEATURE_GITHUB_BUILD: true
      FEATURE_GITLAB_BUILD: true
      FEATURE_MAILING: false
      FEATURE_PROXY_STORAGE: true
      FEATURE_REPO_MIRROR: true
      FEATURE_SECURITY_NOTIFICATIONS: true
      FEATURE_SECURITY_SCANNER: true
      FEATURE_STORAGE_REPLICATION: false
      ........
      PREFERRED_URL_SCHEME: https
      REGISTRY_TITLE: Red Hat Quay
      REGISTRY_TITLE_SHORT: Red Hat Quay
      REPO_MIRROR_INTERVAL: 30
      REPO_MIRROR_TLS_VERIFY: true
      SERVER_HOSTNAME: quayregistry-quay-quay-enterprise.apps.whu413gcp0.qe.gcp.devcluster.openshift.com
      SETUP_COMPLETE: true
      SUPER_USERS:
      - whuquay
      TAG_EXPIRATION_OPTIONS:
      - 2w
      TEAM_RESYNC_STALE_TIME: 60m
      TESTING: false
      USER_EVENTS_REDIS:
        host: quayregistry-quay-redis
        port: 6379
      USERFILES_LOCATION: default
      USERFILES_PATH: userfiles
       
      
      $ oc get quayregistry quayregistry -o  yaml
      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: quayregistry
        namespace: quay-enterprise
        ......
      spec:
        components:
        - kind: tls
          managed: false
        - kind: quay
          managed: true
          overrides:
            env:
            - name: DEBUGLOG
              value: "true"
        - kind: postgres
          managed: true
        - kind: clair
          managed: true
        - kind: redis
          managed: true
        - kind: horizontalpodautoscaler
          managed: false
        - kind: objectstorage
          managed: true
        - kind: route
          managed: true
        - kind: mirror
          managed: true
        - kind: monitoring
          managed: false
        - kind: clairpostgres
          managed: true
        configBundleSecret: config-bundle-secret-20230426-135936

      2. upload docker file to trigger build process

       

      Actual results:

      After click upload button, quay console went to 500 error page

      p1

      Expected results:

      Docker file should be uploaded successfully and build process goes correctly. 

      Additional info:

      Attach quay pod logs in attachment.

      $ oc get pod
      NAME                                               READY   STATUS      RESTARTS        AGE
      quay-operator.v3.8.6-7fb7944f4d-nrmng              1/1     Running     0               10m
      .......
      quayregistry-quay-app-85899768c5-dhdwg             1/1     Running     0               4m10s
      quayregistry-quay-app-85899768c5-vcmxv             1/1     Running     0               4m40s
      quayregistry-quay-app-upgrade-fczcf                0/1     Completed   0               9m33s

              Unassigned Unassigned
              rhwhu Weihua Hu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: