Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-2547

Bridge operator 3.6 failed to sync openshift namespace with Quay org

XMLWordPrintable

      Description of problem:

      Install bridge operator 3.6 by OLM, create quayintegration CR, check org on quay, bridge operator 3.6 failed to sync openshift namespace with Quay org, the bridge operator pod is always CrashLoopBackOff

       

      Version-Release number of selected component (if applicable):

      quay-bridge-operator-bundle-container-v3.6.0-20

      • Index image v4.5: registry-proxy.engineering.redhat.com/rh-osbs/iib:107499
      • Index image v4.6: registry-proxy.engineering.redhat.com/rh-osbs/iib:107502
      • Index image v4.7: registry-proxy.engineering.redhat.com/rh-osbs/iib:107505
      • Index image v4.8: registry-proxy.engineering.redhat.com/rh-osbs/iib:107508
      • Index image v4.9: registry-proxy.engineering.redhat.com/rh-osbs/iib:107511

      quay-bridge-operator-container-v3.6.0-23

      • registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-bridge-operator-rhel8@sha256:edbe00991212746296fe713b9a88c97d074de4e94134584b7e9143f269a5101c

       

      How reproducible:

       always

      Steps to Reproduce:
      1. Install bridge operator 3.6 and perform openshift build, following https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-44659
      2. Check org and repo in Quay UI

      Actual results:

      failed to create org in Quay, the bridge operator pod is always CrashLoopBackOff

      Expected results:

       Should sync up the openshift namespace with org in QUay

      Additional info:

      $ oc describe pod -n openshift-operators
Name:         quay-bridge-operator-controller-manager-6ddb58599c-b8jjz
Namespace:    openshift-operators
Priority:     0
Node:         ip-10-0-196-156.us-east-2.compute.internal/10.0.196.156
Start Time:   Wed, 15 Sep 2021 18:59:11 +0800
Labels:       control-plane=controller-manager
              pod-template-hash=6ddb58599c
Annotations:  alm-examples:
                [
                  {
                    "apiVersion": "quay.redhat.com/v1",
                    "kind": "QuayIntegration",
                    "metadata": {
                      "name": "quay"
                    },
                    "spec": {
                      "clusterID": "openshift",
                      "credentialsSecret": {
                        "name": "quay-credentials",
                        "namespace": "openshift-operators"
                      },
                      "insecureRegistry": false
                    }
                  }
                ]
              capabilities: Full Lifecycle
              categories: OpenShift Optional
              containerImage: registry.redhat.io/quay/quay-bridge-operator-rhel8@sha256:edbe00991212746296fe713b9a88c97d074de4e94134584b7e9143f269a5101c
              createdAt: 2021-09-08 16:09 UTC
              description: Enhance OCP using Red Hat Quay container registry
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.131.0.14"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.131.0.14"
                    ],
                    "default": true,
                    "dns": {}
                }]
              olm.operatorGroup: global-operators
              olm.operatorNamespace: openshift-operators
              olm.skipRange: >=3.5.x <3.6.0
              olm.targetNamespaces: 
              olmcahash: bcb9d2b0aa3d33cb540c91f7cd0da8aa67d8e27d82acfdb26249783b4de35259
              openshift.io/scc: restricted
              operatorframework.io/properties:
                {"properties":[{"type":"olm.package","value":{"packageName":"quay-bridge-operator","version":"3.6.0"}},{"type":"olm.gvk","value":{"group":...
              operators.openshift.io/infrastructure-features: ["disconnected"]
              operators.operatorframework.io/builder: operator-sdk-v1.9.0+git
              operators.operatorframework.io/project_layout: go.kubebuilder.io/v3
              repository: https://github.com/quay/quay-bridge-operator
Status:       Running
IP:           10.131.0.14
IPs:
  IP:           10.131.0.14
Controlled By:  ReplicaSet/quay-bridge-operator-controller-manager-6ddb58599c
Containers:
  kube-rbac-proxy:
    Container ID:  cri-o://dfd3174120259d987c612ba21b27afc58c83a4a8eb87027763cc33162305d3d6
    Image:         registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:deb80f8e208b0133a91fb62909eec014d6245ea7fa8f0040411d019a545bd43a
    Image ID:      registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:4f0fb136c17a69dfa3e09b9a519de88ba0acdb088c941a747a3a2771b7452193
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    State:          Running
      Started:      Wed, 15 Sep 2021 18:59:18 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      OPERATOR_CONDITION_NAME:  quay-bridge-operator.v3.6.0
    Mounts:
      /apiserver.local.config/certificates from apiservice-cert (rw)
      /tmp/k8s-webhook-server/serving-certs from webhook-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n56xg (ro)
  manager:
    Container ID:  cri-o://3423379eb0aae2fea6f205fc40701b28f73d85eed1b0abf8584ec5538aeddf0d
    Image:         registry.redhat.io/quay/quay-bridge-operator-rhel8@sha256:edbe00991212746296fe713b9a88c97d074de4e94134584b7e9143f269a5101c
    Image ID:      registry.redhat.io/quay/quay-bridge-operator-rhel8@sha256:9fb4e5b63319af459c4b44c71197ef4c7a6e8811fe93dd2af45923b3b0b94659
    Port:          9443/TCP
    Host Port:     0/TCP
    Command:
      /manager
    Args:
      --health-probe-bind-address=:8081
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 15 Sep 2021 19:18:47 +0800
      Finished:     Wed, 15 Sep 2021 19:19:12 +0800
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      OPERATOR_CONDITION_NAME:  quay-bridge-operator.v3.6.0
    Mounts:
      /apiserver.local.config/certificates from apiservice-cert (ro)
      /tmp/k8s-webhook-server/serving-certs from webhook-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n56xg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  apiservice-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quay-bridge-operator-controller-manager-service-cert
    Optional:    false
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quay-bridge-operator-controller-manager-service-cert
    Optional:    false
  kube-api-access-n56xg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       23m   default-scheduler  Successfully assigned openshift-operators/quay-bridge-operator-controller-manager-6ddb58599c-b8jjz to ip-10-0-196-156.us-east-2.compute.internal
  Normal   AddedInterface  23m   multus             Add eth0 [10.131.0.14/23] from openshift-sdn
  Normal   Pulling         23m   kubelet            Pulling image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:deb80f8e208b0133a91fb62909eec014d6245ea7fa8f0040411d019a545bd43a"
  Normal   Pulling         23m   kubelet            Pulling image "registry.redhat.io/quay/quay-bridge-operator-rhel8@sha256:edbe00991212746296fe713b9a88c97d074de4e94134584b7e9143f269a5101c"
  Normal   Pulled          23m   kubelet            Successfully pulled image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:deb80f8e208b0133a91fb62909eec014d6245ea7fa8f0040411d019a545bd43a" in 5.129593493s
  Normal   Created         23m   kubelet            Created container kube-rbac-proxy
  Normal   Started         23m   kubelet            Started container kube-rbac-proxy
  Normal   Pulled          23m   kubelet            Successfully pulled image "registry.redhat.io/quay/quay-bridge-operator-rhel8@sha256:edbe00991212746296fe713b9a88c97d074de4e94134584b7e9143f269a5101c" in 6.099194415s
  Warning  ProbeError      11m   kubelet            Readiness probe error: Get "http://10.131.0.14:8081/readyz": read tcp 10.131.0.1:57798->10.131.0.14:8081: read: connection reset by peer
body:
  Warning  Unhealthy   11m  kubelet  Readiness probe failed: Get "http://10.131.0.14:8081/readyz": read tcp 10.131.0.1:57798->10.131.0.14:8081: read: connection reset by peer
  Warning  ProbeError  11m  kubelet  Liveness probe error: Get "http://10.131.0.14:8081/healthz": read tcp 10.131.0.1:57800->10.131.0.14:8081: read: connection reset by peer
body:
  Warning  Unhealthy  11m                   kubelet  Liveness probe failed: Get "http://10.131.0.14:8081/healthz": read tcp 10.131.0.1:57800->10.131.0.14:8081: read: connection reset by peer
  Normal   Created    9m53s (x4 over 23m)   kubelet  Created container manager
  Normal   Started    9m53s (x4 over 23m)   kubelet  Started container manager
  Normal   Pulled     9m53s (x3 over 11m)   kubelet  Container image "registry.redhat.io/quay/quay-bridge-operator-rhel8@sha256:edbe00991212746296fe713b9a88c97d074de4e94134584b7e9143f269a5101c" already present on machine
  Warning  BackOff    3m14s (x33 over 10m)  kubelet  Back-off restarting failed container

       bridge operator pod logs:

      I0915 11:18:48.904082       1 request.go:655] Throttling request took 1.04650814s, request: GET:https://172.30.0.1:443/apis/oauth.openshift.io/v1?timeout=32s
2021-09-15T11:18:50.165Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2021-09-15T11:18:50.166Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/admissionwebhook"}
2021-09-15T11:18:50.166Z	INFO	setup	starting manager
I0915 11:18:50.166545       1 leaderelection.go:243] attempting to acquire leader lease openshift-operators/0111fb36.redhat.com...
2021-09-15T11:18:50.166Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-09-15T11:18:50.166Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
2021-09-15T11:18:50.354Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2021-09-15T11:18:50.354Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
2021-09-15T11:18:50.354Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
I0915 11:19:07.659008       1 leaderelection.go:253] successfully acquired lease openshift-operators/0111fb36.redhat.com
2021-09-15T11:19:07.659Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"openshift-operators","name":"0111fb36.redhat.com","uid":"cb953c84-dcb9-4500-ad8d-4ad06490e99c","apiVersion":"v1","resourceVersion":"42671"}, "reason": "LeaderElection", "message": "quay-bridge-operator-controller-manager-6ddb58599c-b8jjz_7b9d1e37-a89a-45a9-92d5-9330a7a6691a became leader"}
2021-09-15T11:19:07.659Z	INFO	controller-runtime.manager.controller.build	Starting EventSource	{"reconciler group": "build.openshift.io", "reconciler kind": "Build", "source": "kind source: /, Kind="}
2021-09-15T11:19:07.659Z	INFO	controller-runtime.manager.controller.namespace	Starting EventSource	{"reconciler group": "", "reconciler kind": "Namespace", "source": "kind source: /, Kind="}
2021-09-15T11:19:07.659Z	INFO	controller-runtime.manager.controller.quayintegration	Starting EventSource	{"reconciler group": "quay.redhat.com", "reconciler kind": "QuayIntegration", "source": "kind source: /, Kind="}
2021-09-15T11:19:07.759Z	INFO	controller-runtime.manager.controller.build	Starting EventSource	{"reconciler group": "build.openshift.io", "reconciler kind": "Build", "source": "kind source: /, Kind="}
2021-09-15T11:19:07.759Z	INFO	controller-runtime.manager.controller.build	Starting Controller	{"reconciler group": "build.openshift.io", "reconciler kind": "Build"}
2021-09-15T11:19:07.759Z	INFO	controller-runtime.manager.controller.build	Starting workers	{"reconciler group": "build.openshift.io", "reconciler kind": "Build", "worker count": 1}
2021-09-15T11:19:07.759Z	INFO	controller-runtime.manager.controller.namespace	Starting EventSource	{"reconciler group": "", "reconciler kind": "Namespace", "source": "kind source: /, Kind="}
2021-09-15T11:19:07.759Z	INFO	controller-runtime.manager.controller.quayintegration	Starting Controller	{"reconciler group": "quay.redhat.com", "reconciler kind": "QuayIntegration"}
2021-09-15T11:19:07.960Z	INFO	controller-runtime.manager.controller.quayintegration	Starting workers	{"reconciler group": "quay.redhat.com", "reconciler kind": "QuayIntegration", "worker count": 1}
2021-09-15T11:19:07.960Z	INFO	controller-runtime.manager.controller.namespace	Starting Controller	{"reconciler group": "", "reconciler kind": "Namespace"}
2021-09-15T11:19:07.960Z	INFO	controller-runtime.manager.controller.namespace	Starting workers	{"reconciler group": "", "reconciler kind": "Namespace", "worker count": 1}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-ovirt-infra"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-image-registry"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-network-operator"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-config-managed"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-host-network"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-user-workload-monitoring"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-console-operator"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "kube-node-lease"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-ingress"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-console-user-settings"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-sdn"}
2021-09-15T11:19:07.960Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-apiserver-operator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-config-operator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-console"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-cluster-machine-approver"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "kube-system"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-openstack-infra"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-cluster-samples-operator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-cluster-version"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-controller-manager"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-kube-storage-version-migrator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-controller-manager-operator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-kube-storage-version-migrator-operator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-service-ca-operator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-machine-api"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-service-ca"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-dns"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-dns-operator"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-kubevirt-infra"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-kni-infra"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "openshift-insights"}
2021-09-15T11:19:07.961Z	INFO	controllers.NamespaceIntegration	Reconciling Namespace	{"Name": "quay357"}
2021-09-15T11:19:07.970Z	INFO	controllers.QuayIntegration	Updated QuayIntegration status	{"quayintegration": "/quay-registry"}
2021-09-15T11:19:07.970Z	INFO	controllers.QuayIntegration	No changes to QuayIntegration spec, skipping reconciliation	{"quayintegration": "/quay-registry"}

       

      quayintegration CR

      $ oc get quayintegrations.quay.redhat.com  -o yaml
apiVersion: v1
items:
- apiVersion: quay.redhat.com/v1
  kind: QuayIntegration
  metadata:
    creationTimestamp: "2021-09-15T11:11:05Z"
    generation: 1
    name: quay-registry
    resourceVersion: "40046"
    uid: 54c8ac5e-7f39-4567-aaf0-90c7bd2b96ea
  spec:
    clusterID: openshift
    credentialsSecret:
      name: quay-integration
      namespace: openshift-operators
    insecureRegistry: true
    quayHostname: https://registry-quay-quay357.router-default.apps.dyanocp49.qe.devcluster.openshift.com
  status:
    lastUpdate: 2021-09-15 11:13:02.819987933 +0000 UTC
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

              jonathankingfc Jonathan King
              rhn-support-dyan Dongbo Yan
              Dongbo Yan Dongbo Yan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: