Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: None
Affects Version/s: None
Component/s: Controller
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
True
Intelligence Requested:
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

We are demo.redhat.com, proving out MTV mass migration for customer workshops.

Our OCP+Virt clusters are running as guest VMs on OCP+Virt deployed on IBM bare metal.

We are using mostly external Ceph storage.

We have clusters available with similar failure conditions. Will provide access credentials when this issue is picked up.

Background from another cluster's attempt:

MTV deployment of 40 users (40 plans, 160 VMs) began to fail.

OOM errors from the forklift-controller. forklift-controller in CrashLoopBackoff
forklift-controller has two containers: main and inventory.
- main is failing with OOM
- inventory is OK

We are having storage performance issues. Look at what happened to User 30's experience here.

There's a pod, forklift-volume-populator-controller that complained about pvc 'pvc/vmexamples-user30/prime-992a2ced-f6cb-4e67-a445-d806f20656e5' in work queue no longer exists

PVC no longer exists? Why?
PVC plan-user30-vm-54068-r9kbn Error saving claim: Operation cannot be fulfilled on persistentvolumeclaims "plan-user30-vm-54068-r9kbn": the object has been modified; please apply your changes to the latest version and try again
I found the following in Alerts
When running VMs using ODF storage with 'rbd' mounter or 'rbd.csi.ceph.com provisioner', VMs may cause reports of bad crc/signature errors due to certain I/O patterns. Cluster performance can be severely degraded if the number of re-transmissions due to crc errors causes network saturation.100% of the rook-ceph-mgr-external/rook-ceph-mgr-external targets in Namespace NS openshift-storage namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.
the csi-rdbplugin-provisioner is disavowing all knowledge:

and then in the csi-addons :
I0814 13:56:03.470300 1 leaderelection.go:354] lock is held by csi-rbdplugin-provisioner-86679bf8fc-smnv6 and has not yet expired I0814 13:56:03.470331 1 leaderelection.go:255] failed to acquire lease openshift-storage/openshift-storage-rbd-csi-ceph-com-csi-addons
csi-rbdplugin-provisioner-86679bf8fc-q8xtv:
I0814 13:55:14.893726 1 controllerserver-default.go:42] ID: 21 Using default ControllerGetCapabilities
85I0814 13:55:14.893856 1 utils.go:205] ID: 21 GRPC response: {"capabilities":[{"Type":{"Rpc":

{"type":1}

}},{"Type":{"Rpc":

{"type":5}

}},{"Type":{"Rpc":

{"type":7}

}},{"Type":{"Rpc":

{"type":9}

}}]}
86I0814 13:55:44.624873 1 utils.go:198] ID: 22 GRPC call: /identity.Identity/GetCapabilities
87I0814 13:55:44.624910 1 utils.go:199] ID: 22 GRPC request: {}
88I0814 13:55:44.625032 1 utils.go:205] ID: 22 GRPC response: {"capabilities":[{"Type":{"Service":

{"type":1}

}},{"Type":{"ReclaimSpace":

{"type":1}

}},{"Type":{"NetworkFence":

{"type":1}

}},{"Type":{"VolumeReplication":

{"type":1}

}}]}
89I0814 14:01:50.291479 1 utils.go:198] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 GRPC call: /csi.v1.Controller/CreateVolume
90I0814 14:01:50.291670 1 utils.go:199] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 GRPC request: {"capacity_range":

{"required_bytes":22548578304}

,"name":"pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23","parameters":

{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23","csi.storage.k8s.io/pvc/name":"prime-992a2ced-f6cb-4e67-a445-d806f20656e5","csi.storage.k8s.io/pvc/namespace":"vmexamples-user30","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","pool":"ocpv01-tenants","volumeNamePrefix":"ocp4-cluster-vcch9-"}

,"secrets":"*{}stripped{}*","volume_capabilities":\\{"AccessType":{"Block":{}},"access_mode":{"mode":5}}}
91I0814 14:01:50.291910 1 rbd_util.go:1317] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 setting disableInUseChecks: true image features: [object-map layering fast-diff deep-flatten exclusive-lock] mounter: rbd
92I0814 14:01:50.331000 1 omap.go:89] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 got omap values: (pool="ocpv01-tenants", namespace="", name="csi.volumes.default"): map[]
93I0814 14:01:50.334528 1 omap.go:159] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 set omap keys (pool="ocpv01-tenants", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23:8588445b-d9fc-4e74-81e6-2894d583345e])
94I0814 14:01:50.335230 1 omap.go:159] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 set omap keys (pool="ocpv01-tenants", namespace="", name="csi.volume.8588445b-d9fc-4e74-81e6-2894d583345e"): map[csi.imagename:ocp4-cluster-vcch9-8588445b-d9fc-4e74-81e6-2894d583345e csi.volname:pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 csi.volume.owner:vmexamples-user30])
95I0814 14:01:50.335251 1 rbd_journal.go:491] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 generated Volume ID (0001-0011-openshift-storage-0000000000000017-8588445b-d9fc-4e74-81e6-2894d583345e) and image name (ocp4-cluster-vcch9-8588445b-d9fc-4e74-81e6-2894d583345e) for request name (pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23)
96I0814 14:01:50.335294 1 rbd_util.go:425] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 rbd: create ocpv01-tenants/ocp4-cluster-vcch9-8588445b-d9fc-4e74-81e6-2894d583345e size 21504M (features: [exclusive-lock fast-diff deep-flatten layering object-map]) using mon 10.38.85.193:6789,10.38.85.191:6789,10.38.85.192:6789
97I0814 14:01:50.335337 1 rbd_util.go:1565] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 setting image options on ocpv01-tenants/ocp4-cluster-vcch9-8588445b-d9fc-4e74-81e6-2894d583345e
98I0814 14:01:50.356525 1 controllerserver.go:763] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 created image ocpv01-tenants/ocp4-cluster-vcch9-8588445b-d9fc-4e74-81e6-2894d583345e backed for request name pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23
99I0814 14:01:50.367413 1 omap.go:159] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 set omap keys (pool="ocpv01-tenants", namespace="", name="csi.volume.8588445b-d9fc-4e74-81e6-2894d583345e"): map[csi.imageid:719a0fee6be261])
100I0814 14:01:50.367542 1 utils.go:205] ID: 23 Req-ID: pvc-762c7e55-3d70-4c48-bac9-82f7eb834f23 GRPC response: {"volume":{"capacity_bytes":22548578304,"volume_context":

{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"ocp4-cluster-vcch9-8588445b-d9fc-4e74-81e6-2894d583345e","journalPool":"ocpv01-tenants","pool":"ocpv01-tenants","volumeNamePrefix":"ocp4-cluster-vcch9-"}

,"volume_id":"0001-0011-openshift-storage-0000000000000017-8588445b-d9fc-4e74-81e6-2894d583345e"}}