Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: None
Affects Version/s: quay-v3.8.1
Component/s: config-tool, quay
Labels:
- triaged

Blocked:
False
Blocked Reason:
None
Ready:
False
Intelligence Requested:
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

In config-tool web console, set Logs Storage as ELasticsearch.

After setting AWS region of ELasticsearch, the configuration validation passed, but after reconfigure quay, the quay pod will be CrashLoopBackOff.

Version-Release number of selected component (if applicable):

quay 3.8.1
registry.redhat.io/quay/quay-operator-rhel8@sha256:5b549ee6c6bffb3c92390c8959873d7b29cc9d6393a93831f65b8c2236d33690
registry.redhat.io/quay/quay-rhel8@sha256:67132c68b4e5231de5981dc65e1c97cab3b56170534ccc09e2352a4d319f04ee

How reproducible:

always

Steps to Reproduce:
1. set up a workable Elasticsearch

$ curl -u quay:Quay123? https://search-whues-ce5ddmizkfx4qefcv6udfd3ybe.us-east-1.es.amazonaws.com
{
  "name" : "411e67356f5ddadeb2c9c4bd42306651",
  "cluster_name" : "301721915996:whues",
  "cluster_uuid" : "-pyGScPnSGCotUjm9tkZ_A",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2022-09-02T09:02:50.490868Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

2. configure Elasticsearch in config-tool web console
3. input all valid Elasticsearch settings and click "Validate Configuration Changes" button

p1

4. After the validation passed, reconfigure the quay

5. check quay pod status

Actual results:

After reconfigure quay, the quay pod will be CrashLoopBackOff

$ oc get pod
NAME                                               READY   STATUS             RESTARTS       AGE
quay-operator.v3.8.1-b7cb7b869-2mjls               1/1     Running            0              9h
quayregistry-clair-app-685487854f-bml7m            1/1     Running            0              7h41m
quayregistry-clair-app-685487854f-d22cw            1/1     Running            0              7h41m
quayregistry-clair-postgres-d874c76b8-vjpk6        1/1     Running            1 (9h ago)     9h
quayregistry-quay-app-5b98cdc9fc-6nkp5             0/1     CrashLoopBackOff   5 (27s ago)    3m46s
quayregistry-quay-app-79f69f6cfc-sfls7             1/1     Running            0              7h29m
quayregistry-quay-app-79f69f6cfc-snr8v             1/1     Running            0              7h29m
quayregistry-quay-app-upgrade-fqftw                0/1     Completed          0              9h
quayregistry-quay-config-editor-6467b5d9c7-v8tfx   1/1     Running            0              3m46s
quayregistry-quay-database-cb64dbbbc-9hjrj         1/1     Running            0              9h
quayregistry-quay-mirror-5b96f98875-2fdhh          0/1     Error              5 (103s ago)   3m17s
quayregistry-quay-mirror-5b96f98875-2fjdx          0/1     Error              5 (98s ago)    3m17s
quayregistry-quay-redis-775499c8cb-tnwf7           1/1     Running            0              9h

$ oc logs quayregistry-quay-app-5b98cdc9fc-6nkp5    __   __   /  \ /  \     ______   _    _     __   __   __  / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / / / /  / /  \ \  | |  | | | |  | |  / /\ \  \   / \ \  \ \  / /  | |__| | | |__| | / ____ \  | |  \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|   \__/ \__/      \ \__                   \___\ by Red Hat  Build, Store, and Distribute your ContainersStartup timestamp:  Wed Jan 18 12:27:25 UTC 2023Running all default registry services without migration Running init script '/quay-registry/conf/init/certs_install.sh' Installing extra certificates found in /quay-registry/conf/stack/extra_ca_certs directory Running init script '/quay-registry/conf/init/copy_config_files.sh' Running init script '/quay-registry/conf/init/d_validate_config_bundle.sh' Validating Configuration time="2023-01-18T12:27:26Z" level=debug msg="Validating AccessSettings" time="2023-01-18T12:27:26Z" level=debug msg="Validating ActionLogArchiving" time="2023-01-18T12:27:26Z" level=debug msg="Validating AppTokenAuthentication" time="2023-01-18T12:27:26Z" level=debug msg="Validating BitbucketBuildTrigger" time="2023-01-18T12:27:26Z" level=debug msg="Validating BuildManager" time="2023-01-18T12:27:26Z" level=debug msg="Validating Database" time="2023-01-18T12:27:26Z" level=debug msg="Scheme: postgresql" time="2023-01-18T12:27:26Z" level=debug msg="Host: quayregistry-quay-database:5432" time="2023-01-18T12:27:26Z" level=debug msg="Db: quayregistry-quay-database" time="2023-01-18T12:27:26Z" level=debug msg="Params: " time="2023-01-18T12:27:26Z" level=debug msg="Including params " time="2023-01-18T12:27:26Z" level=debug msg="Pinging database at postgresql://quayregistry-quay-database:URLX1Q81p4OGRgmfshUOwYIAc-3TkBBC0EewiscX64lsC5Z10Q3isAIXcEZjinVyOTajSxndRqjQiinO@quayregistry-quay-database:5432/quayregistry-quay-database" plpgsql pg_trgm time="2023-01-18T12:27:26Z" level=debug msg="Validating DistributedStorage" time="2023-01-18T12:27:26Z" level=debug msg="Validating ElasticSearch" time="2023-01-18T12:27:26Z" level=debug msg="Validating Email" time="2023-01-18T12:27:26Z" level=debug msg="Validating GitHubBuildTrigger" time="2023-01-18T12:27:26Z" level=debug msg="Validating GitHubLogin" time="2023-01-18T12:27:26Z" level=debug msg="Validating GitLabBuildTrigger" time="2023-01-18T12:27:26Z" level=debug msg="Validating GoogleLogin" time="2023-01-18T12:27:26Z" level=debug msg="Validating HostSettings" time="2023-01-18T12:27:26Z" level=debug msg="Validating JWTAuthentication" time="2023-01-18T12:27:26Z" level=debug msg="Validating LDAP" time="2023-01-18T12:27:26Z" level=debug msg="Validating OIDC" time="2023-01-18T12:27:26Z" level=debug msg="Validating QuayDocumentation" time="2023-01-18T12:27:26Z" level=debug msg="Validating Redis" time="2023-01-18T12:27:26Z" level=debug msg="Address: quayregistry-quay-redis:6379" time="2023-01-18T12:27:26Z" level=debug msg="Username: " time="2023-01-18T12:27:26Z" level=debug msg="Password Len: 0" time="2023-01-18T12:27:26Z" level=debug msg="Ssl: <nil>" time="2023-01-18T12:27:26Z" level=debug msg="Address: quayregistry-quay-redis:6379" time="2023-01-18T12:27:26Z" level=debug msg="Username: " time="2023-01-18T12:27:26Z" level=debug msg="Password Len: 0" time="2023-01-18T12:27:26Z" level=debug msg="Ssl: <nil>" time="2023-01-18T12:27:26Z" level=debug msg="Validating RepoMirror" time="2023-01-18T12:27:26Z" level=debug msg="Validating SecurityScanner" time="2023-01-18T12:27:26Z" level=debug msg="Validating TeamSyncing" time="2023-01-18T12:27:26Z" level=debug msg="Validating TimeMachine" time="2023-01-18T12:27:26Z" level=debug msg="Validating UserVisibleSettings" +------------------------+-------+--------+ |      Field Group       | Error | Status | +------------------------+-------+--------+ | AccessSettings         | -     | 🟢     | +------------------------+-------+--------+ | ActionLogArchiving     | -     | 🟢     | +------------------------+-------+--------+ | AppTokenAuthentication | -     | 🟢     | +------------------------+-------+--------+ | BitbucketBuildTrigger  | -     | 🟢     | +------------------------+-------+--------+ | BuildManager           | -     | 🟢     | +------------------------+-------+--------+ | Database               | -     | 🟢     | +------------------------+-------+--------+ | DistributedStorage     | -     | 🟢     | +------------------------+-------+--------+ | ElasticSearch          | -     | 🟢     | +------------------------+-------+--------+ | Email                  | -     | 🟢     | +------------------------+-------+--------+ | GitHubBuildTrigger     | -     | 🟢     | +------------------------+-------+--------+ | GitHubLogin            | -     | 🟢     | +------------------------+-------+--------+ | GitLabBuildTrigger     | -     | 🟢     | +------------------------+-------+--------+ | GoogleLogin            | -     | 🟢     | +------------------------+-------+--------+ | HostSettings           | -     | 🟢     | +------------------------+-------+--------+ | JWTAuthentication      | -     | 🟢     | +------------------------+-------+--------+ | LDAP                   | -     | 🟢     | +------------------------+-------+--------+ | OIDC                   | -     | 🟢     | +------------------------+-------+--------+ | QuayDocumentation      | -     | 🟢     | +------------------------+-------+--------+ | Redis                  | -     | 🟢     | +------------------------+-------+--------+ | RepoMirror             | -     | 🟢     | +------------------------+-------+--------+ | SecurityScanner        | -     | 🟢     | +------------------------+-------+--------+ | TeamSyncing            | -     | 🟢     | +------------------------+-------+--------+ | TimeMachine            | -     | 🟢     | +------------------------+-------+--------+ | UserVisibleSettings    | -     | 🟢     | +------------------------+-------+--------+ Running init script '/quay-registry/conf/init/nginx_conf_create.sh' Running init script '/quay-registry/conf/init/supervisord_conf_create.sh' Running init script '/quay-registry/conf/init/zz_boot.sh' Traceback (most recent call last):   File "/quay-registry/./boot.py", line 15, in <module>     from app import app   File "/quay-registry/app.py", line 341, in <module>     logs_model.configure(app.config)   File "/quay-registry/data/logs_model/__init__.py", line 65, in configure     logs_model.initialize(_LOG_MODELS[model_name](**model_config))   File "/quay-registry/data/logs_model/document_logs_model.py", line 151, in __init__     self._es_client = configure_es(**elasticsearch_config)   File "/quay-registry/data/logs_model/elastic_logs.py", line 284, in configure_es     es_client._initialize()   File "/quay-registry/data/logs_model/elastic_logs.py", line 164, in _initialize     self._client.indices.get_template(self._index_prefix)   File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped     return func(*args, params=params, headers=headers, **kwargs)   File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 656, in get_template     return self.transport.perform_request(   File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 355, in perform_request     status, headers_response, data = connection.perform_request(   File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 173, in perform_request     self._raise_error(response.status_code, raw_data)   File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 243, in _raise_error     raise HTTP_EXCEPTIONS.get(status_code, TransportError)( elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, '{"message":"The security token included in the request is invalid."}')

Expected results:

The quay pod should be running successfully.

Additional info:

If keeping other valid settings and remove AWS region setting , after reconfigure quay, all pods work well.

p2

$ oc get pod
NAME                                               READY   STATUS      RESTARTS     AGE
quay-operator.v3.8.1-b7cb7b869-2mjls               1/1     Running     0            9h
quayregistry-clair-app-685487854f-bml7m            1/1     Running     0            7h56m
quayregistry-clair-app-685487854f-d22cw            1/1     Running     0            7h56m
quayregistry-clair-postgres-d874c76b8-vjpk6        1/1     Running     1 (9h ago)   9h
quayregistry-quay-app-6cfc67797-96knl              1/1     Running     0            3m13s
quayregistry-quay-app-6cfc67797-szlrv              1/1     Running     0            3m59s
quayregistry-quay-app-upgrade-fqftw                0/1     Completed   0            9h
quayregistry-quay-config-editor-6cc6987b45-sl4jt   1/1     Running     0            3m59s
quayregistry-quay-database-cb64dbbbc-9hjrj         1/1     Running     0            9h
quayregistry-quay-mirror-5cfbdd6bd7-k45bq          1/1     Running     0            3m58s
quayregistry-quay-mirror-5cfbdd6bd7-t8kk4          1/1     Running     0            3m58s
quayregistry-quay-redis-775499c8cb-tnwf7           1/1     Running     0            9h