Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-10873

Mirror pods will be terminated and cannot be restarted after the securityContext is overwritten

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • quay-v3.17.0
    • quay
    • False
    • Hide

      None

      Show
      None
    • False

      Description of problem:

      Quay 3.17.0  harden quay container security,  set the securityContext as the following value

      "securityContext": {
    "allowPrivilegeEscalation": false,
    "capabilities": {
      "add": [
        "NET_BIND_SERVICE"
      ],
      "drop": [
        "ALL"
      ]
    },
    "runAsNonRoot": true
  }

      However, if attempt to overwrite the securityContext of both the quay and mirror pods, the quay pod will block the overwrite operation and continue running. The mirror pod, on the other hand, will be terminated and cannot be restarted after the overwrite is rejected.

      Version information

      ------------------------------
registry.redhat.io/quay/quay-operator-rhel9@sha256:5544de86a5745eb68f96b28238a13ebd15baa095383e5edbe83d7ad5fccaa40c
------------------------------
registry.redhat.io/quay/quay-rhel9@sha256:e4d4de5fe4e4edaa4ef9215b945f6cc1c9c975dff273e0741551cc15cf424d23
------------------------------
$ oc get quayregistry -o jsonpath='{.items[0].status.currentVersion}'  -n quay-enterprise
3.17.0

      Steps to reproduce:

      1. Deploy quay 3.17.0 in a OCP cluster
      2. After all pods are running, override the securityContext of quay component

      $ oc patch quayregistry quayregistry -n quay-enterprise --type='merge' -p'{"spec":{"components":[{"kind":"quay","managed":true,"overrides":{"securityContext":{"allowPrivilegeEscalation":false,"runAsNonRoot":true,"capabilities":{"drop":["ALL"],"add":["NET_BIND_SERVICE","CHOWN","DAC_OVERRIDE"]}}}}]}}'quayregistry.quay.redhat.com/quayregistry patched

      3. After all pods are running again, override the securityContext of mirror component

      $ oc patch quayregistry quayregistry -n quay-enterprise --type='merge' -p'{"spec":{"components":[{"kind":"mirror","managed":true,"overrides":{"securityContext":{"allowPrivilegeEscalation":false,"runAsNonRoot":true,"capabilities":{"drop":["ALL"],"add":["NET_BIND_SERVICE","CHOWN","DAC_OVERRIDE"]}}}}]}}'quayregistry.quay.redhat.com/quayregistry patched

      Actual results:

      Mirror pods will be terminated and cannot be restarted after the securityContext is overwritten 

       % oc get pod
NAME                                           READY   STATUS      RESTARTS      AGE
quay-operator.v3.17.0-55d9bbb966-6k6q7         1/1     Running     0             105m
quayregistry-clair-app-656765df46-5twj2        1/1     Running     0             15m
quayregistry-clair-app-656765df46-ghzgz        1/1     Running     0             14m
quayregistry-clair-postgres-64bd67fd8d-9r4ch   1/1     Running     1 (63m ago)   63m
quayregistry-quay-app-5659c9559-fbmnb          1/1     Running     0             15m
quayregistry-quay-app-5659c9559-lgswn          1/1     Running     0             13m
quayregistry-quay-app-upgrade-crn5k            0/1     Completed   2             63m
quayregistry-quay-database-55df4bdd6b-8d25d    1/1     Running     0             63m
quayregistry-quay-redis-5c4748887c-fct6p       1/1     Running     0             15m

       

      Expected results

      Mirror pods should continue to work after the overwrite is rejected. Like what the quay pods do.

      Additional info:

      After override mirror component securityContext. got below information.

      Get the whole operator logs in quay-operator.v3.17.0-55d9bbb966-6k6q7

      % oc get quayregistry quayregistry -n quay-enterprise -o jsonpath='{.spec}' | jq .
{
  "components": [
    {
      "kind": "mirror",
      "managed": true,
      "overrides": {
        "securityContext": {
          "allowPrivilegeEscalation": false,
          "capabilities": {
            "add": [
              "NET_BIND_SERVICE",
              "CHOWN",
              "DAC_OVERRIDE"
            ],
            "drop": [
              "ALL"
            ]
          },
          "runAsNonRoot": true
        }
      }
    },
    {
      "kind": "quay",
      "managed": true
    },
......... 
       % oc get quayregistry quayregistry -n quay-enterprise -o jsonpath='{.status.conditions}' | jq .
[
  {
    "lastTransitionTime": "2026-03-10T06:11:19Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Horizontal pod autoscaler found",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentHPAReady"
  },
  {
    "lastTransitionTime": "2026-03-10T05:40:35Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Route admitted",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentRouteReady"
  },
  {
    "lastTransitionTime": "2026-03-10T05:40:23Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Monitoring not managed by the operator",
    "reason": "ComponentNotManaged",
    "status": "True",
    "type": "ComponentMonitoringReady"
  },
  {
    "lastTransitionTime": "2026-03-10T05:41:23Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Deployment quayregistry-quay-database healthy",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentPostgresReady"
  },
  {
    "lastTransitionTime": "2026-03-10T05:40:23Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Object storage not managed by the operator",
    "reason": "ComponentNotManaged",
    "status": "True",
    "type": "ComponentObjectStorageReady"
  },
  {
    "lastTransitionTime": "2026-03-10T06:12:21Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Clair component healthy",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentClairReady"
  },
  {
    "lastTransitionTime": "2026-03-10T05:41:23Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Deployment quayregistry-clair-postgres healthy",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentClairPostgresReady"
  },
  {
    "lastTransitionTime": "2026-03-10T06:29:19Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Using cluster wildcard certs",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentTLSReady"
  },
  {
    "lastTransitionTime": "2026-03-10T05:40:49Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Deployment quayregistry-quay-redis healthy",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentRedisReady"
  },
  {
    "lastTransitionTime": "2026-03-10T06:31:25Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Quay component healthy",
    "reason": "ComponentReady",
    "status": "True",
    "type": "ComponentQuayReady"
  },
  {
    "lastTransitionTime": "2026-03-10T06:11:19Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Deployment quayregistry-quay-mirror has zero replicas available",
    "reason": "ComponentNotReady",
    "status": "False",
    "type": "ComponentMirrorReady"
  },
  {
    "lastTransitionTime": "2026-03-10T06:10:55Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "Some components are not ready",
    "reason": "ComponentNotReady",
    "status": "False",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2026-03-10T05:41:29Z",
    "lastUpdateTime": "2026-03-10T05:41:29Z",
    "message": "All registry components created",
    "reason": "ComponentsCreationSuccess",
    "status": "True",
    "type": "ComponentsCreated"
  },
  {
    "lastTransitionTime": "2026-03-10T06:36:41Z",
    "lastUpdateTime": "2026-03-10T06:36:41Z",
    "message": "All objects created/updated successfully",
    "reason": "ComponentsCreationSuccess",
    "status": "False",
    "type": "RolloutBlocked"
  }
]

      % oc get deployment -n quay-enterprise quayregistry-quay-mirror -o json | jq '.spec.template.spec.containers[] | {name: .name, securityContext: .securityContext}'

{
  "name": "quay-mirror",
  "securityContext": {
    "allowPrivilegeEscalation": false,
    "capabilities": {
      "add": [
        "NET_BIND_SERVICE",
        "CHOWN",
        "DAC_OVERRIDE"
      ],
      "drop": [
        "ALL"
      ]
    },
    "runAsNonRoot": true
  }
}

      oc get events -n quay-enterprise --sort-by='.lastTimestamp'
......
81s         Warning   FailedCreate                   replicaset/quayregistry-quay-mirror-6dbf6dbb9f         Error creating: pods "quayregistry-quay-mirror-6dbf6dbb9f-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "CHOWN": capability may not be added, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "DAC_OVERRIDE": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "CHOWN": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "DAC_OVERRIDE": capability may not be added, provider "restricted-v3": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
60s         Normal    ComponentsCreationSuccess      quayregistry/quayregistry                              All objects created/updated successfully
53s         Warning   FailedGetResourceMetric        horizontalpodautoscaler/quayregistry-quay-mirror       failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
28s         Warning   Unhealthy                      pod/quayregistry-quay-app-5659c9559-lgswn              Startup probe failed: HTTP probe failed with statuscode: 404
16s         Warning   FailedMount                    pod/quayregistry-quay-app-54dd8666b6-fp2hz             MountVolume.SetUp failed for volume "config" : secret "quayregistry-quay-config-secret-bk62dgmgff" not found
13s         Normal    ScalingReplicaSet              deployment/quayregistry-quay-app                       Scaled down replica set quayregistry-quay-app-54dd8666b6 from 1 to 0
13s         Normal    Killing                        pod/quayregistry-quay-app-54dd8666b6-fp2hz             Stopping container quay-app
13s         Normal    SuccessfulDelete               replicaset/quayregistry-quay-app-54dd8666b6            Deleted pod: quayregistry-quay-app-54dd8666b6-fp2hz
8s          Warning   Unhealthy                      pod/quayregistry-quay-app-54dd8666b6-fp2hz             Readiness probe failed: Get "http://10.129.2.35:8080/health/instance": dial tcp 10.129.2.35:8080: connect: connection refused

        1. quay-operator.v3.17.0-55d9bbb966-6k6q7
          1.90 MB

              marckok Marcus Kok
              rhwhu Weihua Hu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: