Loading...

XML

Word

Printable

Type: Bug
Resolution: Won't Do
Priority: Major
Fix Version/s: quay-v3.4.1
Affects Version/s: None
Component/s: quay-operator
Labels:
- triaged

Blocked:
False
Ready:
False
Release Note Text:
Undefined
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

On cluster-bot cluster, the following ObjectBucketClaim has been created

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:   annotations:     quay-buildmanager-hostname: >-
      example-quay-builder-default.apps.ci-ln-rl8w7d2-f76d1.origin-ci-int-gce.dev.openshift.com
    quay-managed-fieldgroups: 'SecurityScanner,Database,DistributedStorage,Redis,HostSettings,RepoMirror'
    quay-operator-service-endpoint: 'http://quay-operator.openshift-operators:7071'
    quay-registry-hostname: ''
    quay-version: vader
  selfLink: >-
    /apis/objectbucket.io/v1alpha1/namespaces/default/objectbucketclaims/example-quay-datastore
  resourceVersion: '56791'
  name: example-quay-datastore
  uid: e8805bec-f2fb-4981-8d1a-54ab45c000da
  creationTimestamp: '2021-02-09T12:55:34Z'
  generation: 3
  managedFields:     - apiVersion: objectbucket.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:quay-buildmanager-hostname': {}
            'f:quay-managed-fieldgroups': {}
            'f:quay-operator-service-endpoint': {}
            'f:quay-registry-hostname': {}
            'f:quay-version': {}
          'f:ownerReferences':
            'k:{"uid":"77fc8344-6bed-4f13-9bc5-599df541d791"}':
              .: {}
              'f:apiVersion': {}
              'f:kind': {}
              'f:name': {}
              'f:uid': {}
        'f:spec':
          'f:ObjectBucketName': {}
          'f:additionalConfig':
            'f:bucketclass': {}
          'f:bucketName': {}
          'f:generateBucketName': {}
          'f:storageClassName': {}
      manager: quay-operator
      operation: Apply
      time: '2021-02-09T12:55:39Z'
    - apiVersion: objectbucket.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:finalizers':
            .: {}
            'v:"objectbucket.io/finalizer"': {}
          'f:labels':
            'f:app': {}
            'f:bucket-provisioner': {}
            'f:noobaa-domain': {}
        'f:status':
          .: {}
          'f:phase': {}
      manager: noobaa-operator
      operation: Update
      time: '2021-02-09T12:55:39Z'
  namespace: default
  ownerReferences:     - apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      name: example
      uid: 77fc8344-6bed-4f13-9bc5-599df541d791
  finalizers:     - objectbucket.io/finalizer
  labels:     app: noobaa
    bucket-provisioner: openshift-storage.noobaa.io-obc
    noobaa-domain: openshift-storage.noobaa.io
spec:   ObjectBucketName: ''
  additionalConfig:     bucketclass: noobaa-default-bucket-class
  bucketName: ''
  generateBucketName: quay-datastore
  storageClassName: openshift-storage.noobaa.io
status:   phase: Pending

The status is pending, however it seems all NooBaa components are up and running:

oc get noobaa -n openshift-storage
NAME     MGMT-ENDPOINTS                S3-ENDPOINTS                  IMAGE                                                                                                            PHASE   AGE
noobaa   ["https://10.0.32.6:31986"]   ["https://10.0.32.6:31023"]   registry.redhat.io/ocs4/mcg-core-rhel8@sha256:cef0031fe36242679707fa6660e967d28f431f43827b3b9ebfa185f4cc02b54a   Ready   28m

oc get deploy
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
noobaa-endpoint        1/1     1            1           16m
noobaa-operator        1/1     1            1           19m
ocs-metrics-exporter   1/1     1            1           19m
ocs-operator           1/1     1            1           19m
rook-ceph-operator     1/1     1            1           19m

Trying to create the QuayRegistry CR in the default namespace ends up with the CR getting stuck in an state that does not point to the root cause but rather a side effect (database migrations pending)

oc describe quayregistry example
Name:         example
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  quay.redhat.com/v1
Kind:         QuayRegistry
Metadata:
  Creation Timestamp:  2021-02-09T12:55:30Z
  Generation:          2
  Managed Fields:
    API Version:  quay.redhat.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:components:
    Manager:      Mozilla
    Operation:    Update
    Time:         2021-02-09T12:55:30Z
    API Version:  quay.redhat.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:quay.redhat.com/object-storage-initialized:
          f:quay.redhat.com/router-canonical-hostname:
          f:quay.redhat.com/supports-object-storage:
          f:quay.redhat.com/supports-routes:
      f:spec:
        f:configBundleSecret:
      f:status:
        .:
        f:conditions:
        f:configEditorCredentialsSecret:
        f:configEditorEndpoint:
        f:lastUpdated:
    Manager:         manager
    Operation:       Update
    Time:            2021-02-09T12:55:41Z
  Resource Version:  56833
  Self Link:         /apis/quay.redhat.com/v1/namespaces/default/quayregistries/example
  UID:               77fc8344-6bed-4f13-9bc5-599df541d791
Spec:
  Components:
    Kind:                clair
    Managed:             true
    Kind:                postgres
    Managed:             true
    Kind:                objectstorage
    Managed:             true
    Kind:                redis
    Managed:             true
    Kind:                horizontalpodautoscaler
    Managed:             true
    Kind:                route
    Managed:             true
    Kind:                mirror
    Managed:             true
  Config Bundle Secret:  example-config-bundle-bqkcs
Status:
  Conditions:
    Last Transition Time:            2021-02-09T12:55:39Z
    Last Update Time:                2021-02-09T12:55:39Z
    Message:                         all objects created/updated successfully
    Reason:                          ComponentsCreationSuccess
    Status:                          False
    Type:                            RolloutBlocked
    Last Transition Time:            2021-02-09T12:55:40Z
    Last Update Time:                2021-02-09T12:55:40Z
    Message:                         running database migrations
    Reason:                          MigrationsInProgress
    Status:                          False
    Type:                            Available
  Config Editor Credentials Secret:  example-quay-config-editor-credentials-7t8bkk545d
  Config Editor Endpoint:            https://example-quay-config-editor-default.apps.ci-ln-rl8w7d2-f76d1.origin-ci-int-gce.dev.openshift.com
  Last Updated:                      2021-02-09 12:55:40.408692189 +0000 UTC
Events:
  Type    Reason                     Age                From                     Message
  ----    ------                     ----               ----                     -------
  Normal  ComponentsCreationSuccess  16m (x2 over 16m)  quayregistry-controller  all objects created/updated successfully
  Normal  MigrationsInProgress       16m                quayregistry-controller  running database migrations

Looking at the resources the pods that require a database seem to have failed:

oc get deploy
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
example-clair-app            0/1     1            0           17m
example-clair-postgres       0/1     1            0           17m
example-quay-app             0/0     0            0           17m
example-quay-app-upgrade     0/1     1            0           17m
example-quay-config-editor   1/1     1            1           17m
example-quay-database        0/1     1            0           17m
example-quay-mirror          0/1     1            0           17m
example-quay-redis           1/1     1            1           17m

oc get pods
NAME                                          READY   STATUS                  RESTARTS   AGE
example-clair-app-5b6d8d7578-fm9rz            0/1     Running                 5          17m
example-clair-app-69db968578-bk8z4            0/1     CrashLoopBackOff        5          17m
example-clair-postgres-c8d985bd8-n769c        0/1     CrashLoopBackOff        8          17m
example-quay-app-upgrade-6df4b5fbcd-rxkhm     0/1     Init:CrashLoopBackOff   5          17m
example-quay-config-editor-5d9c484f54-rfl96   1/1     Running                 0          17m
example-quay-database-585869f7d6-4g7pf        0/1     CrashLoopBackOff        7          17m
example-quay-mirror-6f58bfdb9d-9wj95          0/1     Init:CrashLoopBackOff   5          17m
example-quay-postgres-init-nlvgh              0/1     Completed               0          17m
example-quay-postgres-init-np9km              0/1     Completed               0          17m
example-quay-redis-cc9fbd7c7-tv62j            1/1     Running                 0          17m

oc get events
LAST SEEN   TYPE      REASON                                       OBJECT                                             MESSAGE
128m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-0            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-0 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-0 in Controller
127m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-master-0            openshift-sdn done initializing node networking.
125m        Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-master-0            Setting node ci-ln-rl8w7d2-f76d1-pvzh7-master-0, currentConfig rendered-master-aefffb8b0fb6ce978d32cab905fc1b0c to Done
121m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-0            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-0 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-0 in Controller
102m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-0            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-0 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-0 in Controller
128m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-1            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-1 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-1 in Controller
127m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-master-1            openshift-sdn done initializing node networking.
125m        Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-master-1            Setting node ci-ln-rl8w7d2-f76d1-pvzh7-master-1, currentConfig rendered-master-aefffb8b0fb6ce978d32cab905fc1b0c to Done
121m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-1            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-1 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-1 in Controller
102m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-1            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-1 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-1 in Controller
128m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-2            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-2 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-2 in Controller
127m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-master-2            openshift-sdn done initializing node networking.
125m        Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-master-2            Setting node ci-ln-rl8w7d2-f76d1-pvzh7-master-2, currentConfig rendered-master-aefffb8b0fb6ce978d32cab905fc1b0c to Done
121m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-2            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-2 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-2 in Controller
102m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-master-2            Node ci-ln-rl8w7d2-f76d1-pvzh7-master-2 event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-master-2 in Controller
120m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Starting kubelet.
120m        Normal    NodeHasSufficientMemory                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm status is now: NodeHasSufficientMemory
120m        Normal    NodeHasNoDiskPressure                        node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm status is now: NodeHasNoDiskPressure
120m        Normal    NodeHasSufficientPID                         node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm status is now: NodeHasSufficientPID
120m        Normal    NodeAllocatableEnforced                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Updated Node Allocatable limit across pods
120m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm in Controller
119m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      openshift-sdn done initializing node networking.
119m        Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Setting node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm, currentConfig rendered-worker-cb7d9c7719fe527aedd8d926054738fa to Done
119m        Normal    NodeReady                                    node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm status is now: NodeReady
102m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-f64xm in Controller
88m         Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Starting kubelet.
88m         Normal    NodeHasSufficientMemory                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl status is now: NodeHasSufficientMemory
88m         Normal    NodeHasNoDiskPressure                        node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl status is now: NodeHasNoDiskPressure
88m         Normal    NodeHasSufficientPID                         node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl status is now: NodeHasSufficientPID
88m         Normal    NodeAllocatableEnforced                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Updated Node Allocatable limit across pods
88m         Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl in Controller
88m         Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      openshift-sdn done initializing node networking.
87m         Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Setting node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl, currentConfig rendered-worker-cb7d9c7719fe527aedd8d926054738fa to Done
87m         Normal    NodeReady                                    node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl status is now: NodeReady
120m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Starting kubelet.
120m        Normal    NodeHasSufficientMemory                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx status is now: NodeHasSufficientMemory
120m        Normal    NodeHasNoDiskPressure                        node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx status is now: NodeHasNoDiskPressure
120m        Normal    NodeHasSufficientPID                         node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx status is now: NodeHasSufficientPID
120m        Normal    NodeAllocatableEnforced                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Updated Node Allocatable limit across pods
120m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx in Controller
119m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      openshift-sdn done initializing node networking.
119m        Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Setting node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx, currentConfig rendered-worker-cb7d9c7719fe527aedd8d926054738fa to Done
119m        Normal    NodeReady                                    node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx status is now: NodeReady
102m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx in Controller
88m         Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Starting kubelet.
88m         Normal    NodeHasSufficientMemory                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd status is now: NodeHasSufficientMemory
88m         Normal    NodeHasNoDiskPressure                        node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd status is now: NodeHasNoDiskPressure
88m         Normal    NodeHasSufficientPID                         node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd status is now: NodeHasSufficientPID
88m         Normal    NodeAllocatableEnforced                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Updated Node Allocatable limit across pods
88m         Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd in Controller
88m         Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      openshift-sdn done initializing node networking.
87m         Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Setting node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd, currentConfig rendered-worker-cb7d9c7719fe527aedd8d926054738fa to Done
87m         Normal    NodeReady                                    node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-j7lhd status is now: NodeReady
120m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Starting kubelet.
120m        Normal    NodeHasSufficientMemory                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj status is now: NodeHasSufficientMemory
120m        Normal    NodeHasNoDiskPressure                        node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj status is now: NodeHasNoDiskPressure
120m        Normal    NodeHasSufficientPID                         node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj status is now: NodeHasSufficientPID
120m        Normal    NodeAllocatableEnforced                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Updated Node Allocatable limit across pods
119m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj in Controller
119m        Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      openshift-sdn done initializing node networking.
119m        Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Setting node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj, currentConfig rendered-worker-cb7d9c7719fe527aedd8d926054738fa to Done
119m        Normal    NodeReady                                    node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj status is now: NodeReady
102m        Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj in Controller
88m         Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Starting kubelet.
88m         Normal    NodeHasSufficientMemory                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf status is now: NodeHasSufficientMemory
88m         Normal    NodeHasNoDiskPressure                        node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf status is now: NodeHasNoDiskPressure
88m         Normal    NodeHasSufficientPID                         node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf status is now: NodeHasSufficientPID
88m         Normal    NodeAllocatableEnforced                      node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Updated Node Allocatable limit across pods
88m         Normal    RegisteredNode                               node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf event: Registered Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf in Controller
88m         Normal    Starting                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      openshift-sdn done initializing node networking.
87m         Normal    NodeDone                                     node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Setting node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf, currentConfig rendered-worker-cb7d9c7719fe527aedd8d926054738fa to Done
87m         Normal    NodeReady                                    node/ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf      Node ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf status is now: NodeReady
17m         Normal    Scheduled                                    pod/example-clair-app-5b6d8d7578-fm9rz             Successfully assigned default/example-clair-app-5b6d8d7578-fm9rz to ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj
17m         Warning   FailedMount                                  pod/example-clair-app-5b6d8d7578-fm9rz             MountVolume.SetUp failed for volume "config" : secret "example-clair-config-secret" not found
17m         Warning   FailedMount                                  pod/example-clair-app-5b6d8d7578-fm9rz             MountVolume.SetUp failed for volume "certs" : secret "example-quay-config-secret-g6k8g7d2fc" not found
17m         Normal    AddedInterface                               pod/example-clair-app-5b6d8d7578-fm9rz             Add eth0 [10.129.2.13/23]
17m         Normal    Pulling                                      pod/example-clair-app-5b6d8d7578-fm9rz             Pulling image "registry.redhat.io/quay/clair-rhel8@sha256:b0b4b270abf3acfbaeb7330ef721fa0573f677e0efafc12cc62274679d496676"
16m         Normal    Pulled                                       pod/example-clair-app-5b6d8d7578-fm9rz             Successfully pulled image "registry.redhat.io/quay/clair-rhel8@sha256:b0b4b270abf3acfbaeb7330ef721fa0573f677e0efafc12cc62274679d496676" in 1m3.158833837s
14m         Normal    Created                                      pod/example-clair-app-5b6d8d7578-fm9rz             Created container clair-app
14m         Normal    Started                                      pod/example-clair-app-5b6d8d7578-fm9rz             Started container clair-app
7m22s       Warning   Unhealthy                                    pod/example-clair-app-5b6d8d7578-fm9rz             Readiness probe failed: dial tcp 10.129.2.13:8080: connect: connection refused
2m21s       Normal    Pulled                                       pod/example-clair-app-5b6d8d7578-fm9rz             Container image "registry.redhat.io/quay/clair-rhel8@sha256:b0b4b270abf3acfbaeb7330ef721fa0573f677e0efafc12cc62274679d496676" already present on machine
17m         Normal    SuccessfulCreate                             replicaset/example-clair-app-5b6d8d7578            Created pod: example-clair-app-5b6d8d7578-fm9rz
17m         Normal    Scheduled                                    pod/example-clair-app-69db968578-bk8z4             Successfully assigned default/example-clair-app-69db968578-bk8z4 to ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl
17m         Warning   FailedMount                                  pod/example-clair-app-69db968578-bk8z4             MountVolume.SetUp failed for volume "certs" : secret "example-quay-config-secret-tcm2bbg8m7" not found
17m         Normal    AddedInterface                               pod/example-clair-app-69db968578-bk8z4             Add eth0 [10.130.2.18/23]
17m         Normal    Pulling                                      pod/example-clair-app-69db968578-bk8z4             Pulling image "registry.redhat.io/quay/clair-rhel8@sha256:b0b4b270abf3acfbaeb7330ef721fa0573f677e0efafc12cc62274679d496676"
16m         Normal    Pulled                                       pod/example-clair-app-69db968578-bk8z4             Successfully pulled image "registry.redhat.io/quay/clair-rhel8@sha256:b0b4b270abf3acfbaeb7330ef721fa0573f677e0efafc12cc62274679d496676" in 19.516859224s
14m         Normal    Created                                      pod/example-clair-app-69db968578-bk8z4             Created container clair-app
14m         Normal    Started                                      pod/example-clair-app-69db968578-bk8z4             Started container clair-app
2m15s       Warning   Unhealthy                                    pod/example-clair-app-69db968578-bk8z4             Readiness probe failed: dial tcp 10.130.2.18:8080: connect: connection refused
12m         Normal    Pulled                                       pod/example-clair-app-69db968578-bk8z4             Container image "registry.redhat.io/quay/clair-rhel8@sha256:b0b4b270abf3acfbaeb7330ef721fa0573f677e0efafc12cc62274679d496676" already present on machine
7m18s       Warning   BackOff                                      pod/example-clair-app-69db968578-bk8z4             Back-off restarting failed container
17m         Normal    SuccessfulCreate                             replicaset/example-clair-app-69db968578            Created pod: example-clair-app-69db968578-bk8z4
17m         Normal    ScalingReplicaSet                            deployment/example-clair-app                       Scaled up replica set example-clair-app-5b6d8d7578 to 1
17m         Normal    ScalingReplicaSet                            deployment/example-clair-app                       Scaled up replica set example-clair-app-69db968578 to 1
17m         Normal    Scheduled                                    pod/example-clair-postgres-c8d985bd8-n769c         Successfully assigned default/example-clair-postgres-c8d985bd8-n769c to ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl
17m         Normal    SuccessfulAttachVolume                       pod/example-clair-postgres-c8d985bd8-n769c         AttachVolume.Attach succeeded for volume "pvc-5ec0ed5a-3afb-425a-9aa1-9aa14c892716"
17m         Normal    AddedInterface                               pod/example-clair-postgres-c8d985bd8-n769c         Add eth0 [10.130.2.19/23]
17m         Normal    Pulling                                      pod/example-clair-postgres-c8d985bd8-n769c         Pulling image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348"
16m         Normal    Pulled                                       pod/example-clair-postgres-c8d985bd8-n769c         Successfully pulled image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348" in 16.015827648s
15m         Normal    Created                                      pod/example-clair-postgres-c8d985bd8-n769c         Created container postgres
15m         Normal    Started                                      pod/example-clair-postgres-c8d985bd8-n769c         Started container postgres
15m         Normal    Pulled                                       pod/example-clair-postgres-c8d985bd8-n769c         Container image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348" already present on machine
114s        Warning   BackOff                                      pod/example-clair-postgres-c8d985bd8-n769c         Back-off restarting failed container
17m         Normal    SuccessfulCreate                             replicaset/example-clair-postgres-c8d985bd8        Created pod: example-clair-postgres-c8d985bd8-n769c
17m         Normal    WaitForFirstConsumer                         persistentvolumeclaim/example-clair-postgres       waiting for first consumer to be created before binding
17m         Normal    ScalingReplicaSet                            deployment/example-clair-postgres                  Scaled up replica set example-clair-postgres-c8d985bd8 to 1
17m         Normal    ProvisioningSucceeded                        persistentvolumeclaim/example-clair-postgres       Successfully provisioned volume pvc-5ec0ed5a-3afb-425a-9aa1-9aa14c892716 using kubernetes.io/gce-pd
17m         Normal    Scheduled                                    pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Successfully assigned default/example-quay-app-upgrade-6df4b5fbcd-rxkhm to ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf
17m         Normal    AddedInterface                               pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Add eth0 [10.131.2.14/23]
17m         Normal    Pulling                                      pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Pulling image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285"
16m         Normal    Pulled                                       pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Successfully pulled image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" in 41.865382737s
6m18s       Normal    Created                                      pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Created container quay-app-upgrade-init
6m18s       Normal    Started                                      pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Started container quay-app-upgrade-init
6m18s       Normal    Pulled                                       pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Container image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" already present on machine
28s         Warning   BackOff                                      pod/example-quay-app-upgrade-6df4b5fbcd-rxkhm      Back-off restarting failed container
17m         Normal    SuccessfulCreate                             replicaset/example-quay-app-upgrade-6df4b5fbcd     Created pod: example-quay-app-upgrade-6df4b5fbcd-rxkhm
17m         Normal    Scheduled                                    pod/example-quay-app-upgrade-77fcf9c474-jwx88      Successfully assigned default/example-quay-app-upgrade-77fcf9c474-jwx88 to ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-9ccqf
17m         Warning   FailedMount                                  pod/example-quay-app-upgrade-77fcf9c474-jwx88      MountVolume.SetUp failed for volume "configvolume" : secret "example-quay-config-secret-g6k8g7d2fc" not found
17m         Normal    AddedInterface                               pod/example-quay-app-upgrade-77fcf9c474-jwx88      Add eth0 [10.131.2.13/23]
17m         Normal    Pulling                                      pod/example-quay-app-upgrade-77fcf9c474-jwx88      Pulling image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285"
16m         Normal    Pulled                                       pod/example-quay-app-upgrade-77fcf9c474-jwx88      Successfully pulled image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" in 47.23533796s
16m         Warning   Failed                                       pod/example-quay-app-upgrade-77fcf9c474-jwx88      Error: cannot find volume "default-token-n899f" to mount into container "quay-app-upgrade-init"
17m         Normal    SuccessfulCreate                             replicaset/example-quay-app-upgrade-77fcf9c474     Created pod: example-quay-app-upgrade-77fcf9c474-jwx88
17m         Normal    SuccessfulDelete                             replicaset/example-quay-app-upgrade-77fcf9c474     Deleted pod: example-quay-app-upgrade-77fcf9c474-jwx88
17m         Normal    ScalingReplicaSet                            deployment/example-quay-app-upgrade                Scaled up replica set example-quay-app-upgrade-77fcf9c474 to 1
17m         Normal    ScalingReplicaSet                            deployment/example-quay-app-upgrade                Scaled down replica set example-quay-app-upgrade-77fcf9c474 to 0
17m         Normal    ScalingReplicaSet                            deployment/example-quay-app-upgrade                Scaled up replica set example-quay-app-upgrade-6df4b5fbcd to 1
17m         Normal    Scheduled                                    pod/example-quay-config-editor-5d9c484f54-rfl96    Successfully assigned default/example-quay-config-editor-5d9c484f54-rfl96 to ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj
17m         Warning   FailedMount                                  pod/example-quay-config-editor-5d9c484f54-rfl96    MountVolume.SetUp failed for volume "config-bundle" : secret "example-quay-config-secret-tcm2bbg8m7" not found
17m         Normal    AddedInterface                               pod/example-quay-config-editor-5d9c484f54-rfl96    Add eth0 [10.129.2.15/23]
17m         Normal    Pulling                                      pod/example-quay-config-editor-5d9c484f54-rfl96    Pulling image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285"
16m         Normal    Pulled                                       pod/example-quay-config-editor-5d9c484f54-rfl96    Successfully pulled image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" in 59.470135508s
16m         Normal    Created                                      pod/example-quay-config-editor-5d9c484f54-rfl96    Created container quay-config-editor
16m         Normal    Started                                      pod/example-quay-config-editor-5d9c484f54-rfl96    Started container quay-config-editor
17m         Normal    SuccessfulCreate                             replicaset/example-quay-config-editor-5d9c484f54   Created pod: example-quay-config-editor-5d9c484f54-rfl96
17m         Normal    Scheduled                                    pod/example-quay-config-editor-6db6fb89b6-rg5bd    Successfully assigned default/example-quay-config-editor-6db6fb89b6-rg5bd to ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx
17m         Warning   FailedMount                                  pod/example-quay-config-editor-6db6fb89b6-rg5bd    MountVolume.SetUp failed for volume "config-bundle" : secret "example-quay-config-secret-g6k8g7d2fc" not found
17m         Normal    AddedInterface                               pod/example-quay-config-editor-6db6fb89b6-rg5bd    Add eth0 [10.131.0.20/23]
17m         Normal    Pulling                                      pod/example-quay-config-editor-6db6fb89b6-rg5bd    Pulling image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285"
16m         Normal    Pulled                                       pod/example-quay-config-editor-6db6fb89b6-rg5bd    Successfully pulled image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" in 52.811281303s
16m         Normal    Created                                      pod/example-quay-config-editor-6db6fb89b6-rg5bd    Created container quay-config-editor
16m         Normal    Started                                      pod/example-quay-config-editor-6db6fb89b6-rg5bd    Started container quay-config-editor
16m         Normal    Killing                                      pod/example-quay-config-editor-6db6fb89b6-rg5bd    Stopping container quay-config-editor
17m         Normal    SuccessfulCreate                             replicaset/example-quay-config-editor-6db6fb89b6   Created pod: example-quay-config-editor-6db6fb89b6-rg5bd
16m         Normal    SuccessfulDelete                             replicaset/example-quay-config-editor-6db6fb89b6   Deleted pod: example-quay-config-editor-6db6fb89b6-rg5bd
17m         Normal    ScalingReplicaSet                            deployment/example-quay-config-editor              Scaled up replica set example-quay-config-editor-6db6fb89b6 to 1
17m         Normal    ScalingReplicaSet                            deployment/example-quay-config-editor              Scaled up replica set example-quay-config-editor-5d9c484f54 to 1
16m         Normal    ScalingReplicaSet                            deployment/example-quay-config-editor              Scaled down replica set example-quay-config-editor-6db6fb89b6 to 0
17m         Normal    Scheduled                                    pod/example-quay-database-585869f7d6-4g7pf         Successfully assigned default/example-quay-database-585869f7d6-4g7pf to ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj
17m         Normal    SuccessfulAttachVolume                       pod/example-quay-database-585869f7d6-4g7pf         AttachVolume.Attach succeeded for volume "pvc-2df311ca-6dd5-481a-aab3-d83548a572d1"
16m         Normal    AddedInterface                               pod/example-quay-database-585869f7d6-4g7pf         Add eth0 [10.129.2.16/23]
14m         Normal    Pulled                                       pod/example-quay-database-585869f7d6-4g7pf         Container image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348" already present on machine
14m         Normal    Created                                      pod/example-quay-database-585869f7d6-4g7pf         Created container postgres
14m         Normal    Started                                      pod/example-quay-database-585869f7d6-4g7pf         Started container postgres
61s         Warning   BackOff                                      pod/example-quay-database-585869f7d6-4g7pf         Back-off restarting failed container
17m         Normal    SuccessfulCreate                             replicaset/example-quay-database-585869f7d6        Created pod: example-quay-database-585869f7d6-4g7pf
17m         Normal    Scheduled                                    pod/example-quay-database-6fc6cddd98-56zvk         Successfully assigned default/example-quay-database-6fc6cddd98-56zvk to ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj
17m         Normal    SuccessfulAttachVolume                       pod/example-quay-database-6fc6cddd98-56zvk         AttachVolume.Attach succeeded for volume "pvc-2df311ca-6dd5-481a-aab3-d83548a572d1"
15m         Warning   FailedMount                                  pod/example-quay-database-6fc6cddd98-56zvk         Unable to attach or mount volumes: unmounted volumes=[postgres-data default-token-n899f], unattached volumes=[postgres-data default-token-n899f]: timed out waiting for the condition
17m         Normal    SuccessfulCreate                             replicaset/example-quay-database-6fc6cddd98        Created pod: example-quay-database-6fc6cddd98-56zvk
17m         Normal    SuccessfulDelete                             replicaset/example-quay-database-6fc6cddd98        Deleted pod: example-quay-database-6fc6cddd98-56zvk
17m         Normal    WaitForFirstConsumer                         persistentvolumeclaim/example-quay-database        waiting for first consumer to be created before binding
17m         Normal    ScalingReplicaSet                            deployment/example-quay-database                   Scaled up replica set example-quay-database-6fc6cddd98 to 1
17m         Normal    ProvisioningSucceeded                        persistentvolumeclaim/example-quay-database        Successfully provisioned volume pvc-2df311ca-6dd5-481a-aab3-d83548a572d1 using kubernetes.io/gce-pd
17m         Normal    ScalingReplicaSet                            deployment/example-quay-database                   Scaled down replica set example-quay-database-6fc6cddd98 to 0
17m         Normal    ScalingReplicaSet                            deployment/example-quay-database                   Scaled up replica set example-quay-database-585869f7d6 to 1
17m         Normal    Scheduled                                    pod/example-quay-mirror-5dcbc89846-brbkn           Successfully assigned default/example-quay-mirror-5dcbc89846-brbkn to ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx
17m         Normal    AddedInterface                               pod/example-quay-mirror-5dcbc89846-brbkn           Add eth0 [10.131.0.19/23]
17m         Normal    Pulling                                      pod/example-quay-mirror-5dcbc89846-brbkn           Pulling image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285"
16m         Normal    Pulled                                       pod/example-quay-mirror-5dcbc89846-brbkn           Successfully pulled image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" in 54.798336626s
16m         Warning   Failed                                       pod/example-quay-mirror-5dcbc89846-brbkn           Error: cannot find volume "default-token-n899f" to mount into container "quay-mirror-init"
17m         Normal    SuccessfulCreate                             replicaset/example-quay-mirror-5dcbc89846          Created pod: example-quay-mirror-5dcbc89846-brbkn
17m         Normal    SuccessfulDelete                             replicaset/example-quay-mirror-5dcbc89846          Deleted pod: example-quay-mirror-5dcbc89846-brbkn
17m         Normal    Scheduled                                    pod/example-quay-mirror-6f58bfdb9d-9wj95           Successfully assigned default/example-quay-mirror-6f58bfdb9d-9wj95 to ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx
17m         Normal    AddedInterface                               pod/example-quay-mirror-6f58bfdb9d-9wj95           Add eth0 [10.131.0.21/23]
17m         Normal    Pulling                                      pod/example-quay-mirror-6f58bfdb9d-9wj95           Pulling image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285"
16m         Normal    Pulled                                       pod/example-quay-mirror-6f58bfdb9d-9wj95           Successfully pulled image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" in 43.597733989s
6m26s       Normal    Created                                      pod/example-quay-mirror-6f58bfdb9d-9wj95           Created container quay-mirror-init
6m26s       Normal    Started                                      pod/example-quay-mirror-6f58bfdb9d-9wj95           Started container quay-mirror-init
6m26s       Normal    Pulled                                       pod/example-quay-mirror-6f58bfdb9d-9wj95           Container image "registry.redhat.io/quay/quay-rhel8@sha256:5ec1c488609362f2ab4dae54b48f87dbc2d378e9525fa852cb69a3a0c545a285" already present on machine
41s         Warning   BackOff                                      pod/example-quay-mirror-6f58bfdb9d-9wj95           Back-off restarting failed container
17m         Normal    SuccessfulCreate                             replicaset/example-quay-mirror-6f58bfdb9d          Created pod: example-quay-mirror-6f58bfdb9d-9wj95
17m         Normal    ScalingReplicaSet                            deployment/example-quay-mirror                     Scaled up replica set example-quay-mirror-5dcbc89846 to 1
17m         Normal    ScalingReplicaSet                            deployment/example-quay-mirror                     Scaled down replica set example-quay-mirror-5dcbc89846 to 0
17m         Normal    ScalingReplicaSet                            deployment/example-quay-mirror                     Scaled up replica set example-quay-mirror-6f58bfdb9d to 1
17m         Normal    Scheduled                                    pod/example-quay-postgres-init-nlvgh               Successfully assigned default/example-quay-postgres-init-nlvgh to ci-ln-rl8w7d2-f76d1-pvzh7-worker-d-6s8cj
17m         Normal    AddedInterface                               pod/example-quay-postgres-init-nlvgh               Add eth0 [10.129.2.14/23]
17m         Normal    Pulling                                      pod/example-quay-postgres-init-nlvgh               Pulling image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348"
16m         Normal    Pulled                                       pod/example-quay-postgres-init-nlvgh               Successfully pulled image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348" in 58.611828894s
16m         Normal    Created                                      pod/example-quay-postgres-init-nlvgh               Created container quay-postgres-init
16m         Normal    Started                                      pod/example-quay-postgres-init-nlvgh               Started container quay-postgres-init
17m         Normal    Scheduled                                    pod/example-quay-postgres-init-np9km               Successfully assigned default/example-quay-postgres-init-np9km to ci-ln-rl8w7d2-f76d1-pvzh7-worker-c-hxrxx
17m         Normal    AddedInterface                               pod/example-quay-postgres-init-np9km               Add eth0 [10.131.0.18/23]
17m         Normal    Pulling                                      pod/example-quay-postgres-init-np9km               Pulling image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348"
17m         Normal    Pulled                                       pod/example-quay-postgres-init-np9km               Successfully pulled image "registry.redhat.io/rhel8/postgresql-10@sha256:572eee6b0e9fd09e1a8d136bcb34b3293648df9e12ea3be49e52ecb3f505e348" in 19.059872928s
17m         Normal    Created                                      pod/example-quay-postgres-init-np9km               Created container quay-postgres-init
17m         Normal    Started                                      pod/example-quay-postgres-init-np9km               Started container quay-postgres-init
17m         Normal    SuccessfulCreate                             job/example-quay-postgres-init                     Created pod: example-quay-postgres-init-np9km
17m         Normal    SuccessfulCreate                             job/example-quay-postgres-init                     Created pod: example-quay-postgres-init-nlvgh
14m         Normal    Completed                                    job/example-quay-postgres-init                     Job completed
17m         Normal    Scheduled                                    pod/example-quay-redis-cc9fbd7c7-tv62j             Successfully assigned default/example-quay-redis-cc9fbd7c7-tv62j to ci-ln-rl8w7d2-f76d1-pvzh7-worker-b-qpscl
17m         Normal    AddedInterface                               pod/example-quay-redis-cc9fbd7c7-tv62j             Add eth0 [10.130.2.17/23]
17m         Normal    Pulling                                      pod/example-quay-redis-cc9fbd7c7-tv62j             Pulling image "registry.redhat.io/rhel8/redis-5@sha256:f670ea6359e62df444b40ba62926b1d64c2ec18910e8396d6bc803615ed97c2b"
17m         Normal    Pulled                                       pod/example-quay-redis-cc9fbd7c7-tv62j             Successfully pulled image "registry.redhat.io/rhel8/redis-5@sha256:f670ea6359e62df444b40ba62926b1d64c2ec18910e8396d6bc803615ed97c2b" in 12.571979418s
17m         Normal    Created                                      pod/example-quay-redis-cc9fbd7c7-tv62j             Created container redis-master
17m         Normal    Started                                      pod/example-quay-redis-cc9fbd7c7-tv62j             Started container redis-master
17m         Normal    SuccessfulCreate                             replicaset/example-quay-redis-cc9fbd7c7            Created pod: example-quay-redis-cc9fbd7c7-tv62j
17m         Normal    ScalingReplicaSet                            deployment/example-quay-redis                      Scaled up replica set example-quay-redis-cc9fbd7c7 to 1
17m         Normal    ComponentsCreationSuccess                    quayregistry/example                               all objects created/updated successfully
17m         Normal    MigrationsInProgress                         quayregistry/example                               running database migrations
<unknown>   Normal    TerminationStart                             namespace/kube-system                              Received signal to terminate, becoming unready, but keeping serving
<unknown>   Normal    TerminationPreShutdownHooksFinished          namespace/kube-system                              All pre-shutdown hooks have been finished
119m        Normal    TerminationStart                             namespace/kube-system                              Received signal to terminate, becoming unready, but keeping serving
119m        Normal    TerminationPreShutdownHooksFinished          namespace/kube-system                              All pre-shutdown hooks have been finished
119m        Normal    TerminationMinimalShutdownDurationFinished   namespace/kube-system                              The minimal shutdown duration of 0s finished
119m        Normal    TerminationStoppedServing                    namespace/kube-system                              Server has stopped listening
118m        Normal    TerminationGracefulTerminationFinished       namespace/kube-system                              All pending requests processed
117m        Normal    TerminationStart                             namespace/kube-system                              Received signal to terminate, becoming unready, but keeping serving
117m        Normal    TerminationPreShutdownHooksFinished          namespace/kube-system                              All pre-shutdown hooks have been finished
117m        Normal    TerminationMinimalShutdownDurationFinished   namespace/kube-system                              The minimal shutdown duration of 0s finished
117m        Normal    TerminationStoppedServing                    namespace/kube-system                              Server has stopped listening
116m        Normal    TerminationGracefulTerminationFinished       namespace/kube-system                              All pending requests processed
104m        Normal    TerminationStart                             namespace/kube-system                              Received signal to terminate, becoming unready, but keeping serving
104m        Normal    TerminationPreShutdownHooksFinished          namespace/kube-system                              All pre-shutdown hooks have been finished
104m        Normal    TerminationMinimalShutdownDurationFinished   namespace/kube-system                              The minimal shutdown duration of 0s finished
104m        Normal    TerminationStoppedServing                    namespace/kube-system                              Server has stopped listening
103m        Normal    TerminationGracefulTerminationFinished       namespace/kube-system                              All pending requests processed
121m        Normal    Status upgrade                               clusteroperator/machine-api                        Progressing towards operator: 4.6.16
121m        Warning   Status degraded                              clusteroperator/machine-api                        Internal error occurred: admission plugin "image.openshift.io/ImagePolicy" failed to complete mutation in 13s
118m        Normal    Status upgrade                               clusteroperator/machine-api                        Progressing towards operator: 4.6.16
127m        Normal    OperatorVersionChanged                       /machine-config                                    clusteroperator/machine-config-operator started a version change from [] to [{operator 4.6.16}]
124m        Normal    OperatorVersionChanged                       /machine-config                                    clusteroperator/machine-config-operator version changed from [] to [{operator 4.6.16}]
125m        Normal    AnnotationChange                             machineconfigpool/master                           Node ci-ln-rl8w7d2-f76d1-pvzh7-master-0 now has machineconfiguration.openshift.io/state=Done
125m        Normal    AnnotationChange                             machineconfigpool/master                           Node ci-ln-rl8w7d2-f76d1-pvzh7-master-2 now has machineconfiguration.openshift.io/state=Done
125m        Normal    AnnotationChange                             machineconfigpool/master                           Node ci-ln-rl8w7d2-f76d1-pvzh7-master-1 now has machineconfiguration.openshift.io/state=Done
125m        Warning   FailedToCreateEndpoint                       endpoints/router-internal-default                  Failed to create endpoint for service openshift-ingress/router-internal-default: endpoints "router-internal-default" already exists

It's unclear why exactly those pods are failing but it seems reasonable to assume it is because NooBaa failed to initialize.

Assignee:: Unassigned

Reporter:: Daniel Messer

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2021/02/09 8:17 AM

Updated:: 2021/03/15 9:36 AM

Resolved:: 2021/03/08 10:10 AM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates