Uploaded image for project: 'Data Foundation Bugs'
  1. Data Foundation Bugs
  2. DFBUGS-851

[IBM Z ] - OSD pods on CLBO state in ODF 4.18

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • odf-4.17
    • odf-4.18
    • ceph
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • s390x
    • ?
    • ?
    • Critical
    • None

      Description of problem - Provide a detailed description of the issue encountered, including logs/command-output snippets and screenshots if the issue is observed in the UI:

      ODF 4.18 deployment is not succeeding on IBM Z due to below error in OSD.

      rook-ceph-osd-0-89c674c8d-ckzmq                                   1/2     CrashLoopBackOff   222 (104s ago)    19h
      rook-ceph-osd-1-8b9d9979b-fkpzm                                   1/2     CrashLoopBackOff   219 (3m19s ago)   19h
      rook-ceph-osd-2-6c97fcccd9-q9fjt                                  1/2     CrashLoopBackOff   217 (4m55s ago)   19h
      
      [root@m1301001 ~]# oc -n openshift-storage logs rook-ceph-osd-0-89c674c8d-ckzmq
      Defaulted container "osd" out of: osd, log-collector, blkdevmapper (init), activate (init), expand-bluefs (init), chown-container-data-dir (init)
      + set -o nounset
      + child_pid=
      + sigterm_received=false
      + trap sigterm SIGTERM
      + child_pid=9448
      + wait 9448
      + ceph-osd --foreground --id 0 --fsid e00734a5-785b-4e3e-816d-852ef0d7e4a0 --setuser ceph --setgroup ceph '--crush-location=root=default host=worker-2-m1301001-lnxero1-boe' --osd-op-num-threads-per-shard=2 --osd-op-num-shards=8 --osd-recovery-sleep=0 --osd-snap-trim-sleep=0 --osd-delete-sleep=0 --bluestore-min-alloc-size=4096 --bluestore-prefer-deferred-size=0 --bluestore-compression-min-blob-size=8192 --bluestore-compression-max-blob-size=65536 --bluestore-max-blob-size=65536 --bluestore-cache-size=3221225472 --bluestore-throttle-cost-per-io=4000 --bluestore-deferred-batch-ops=16 --default-log-to-stderr=true --default-err-to-stderr=true --default-mon-cluster-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-log-to-file=false --default-mon-cluster-log-to-file=false --ms-learn-addr-from-peer=false
      debug 2024-11-14T08:14:20.515+0000 3ff97be5800  0 set uid:gid to 167:167 (ceph:ceph)
      debug 2024-11-14T08:14:20.515+0000 3ff97be5800  0 ceph version 19.2.0-47.el9cp (123a317ae596caa7f6d087fc76fffb6a736e0b5f) squid (stable), process ceph-osd, pid 9448
      debug 2024-11-14T08:14:20.515+0000 3ff97be5800  0 pidfile_write: ignore empty --pid-file
      debug 2024-11-14T08:14:20.515+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
      debug 2024-11-14T08:14:20.515+0000 3ff97be5800  0 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-0/block failed: (22) Invalid argument
      debug 2024-11-14T08:14:20.515+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) open size 1099511627776 (0x10000000000, 1 TiB) block_size 4096 (4 KiB) non-rotational device, discard supported
      debug 2024-11-14T08:14:20.515+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) close
      debug 2024-11-14T08:14:21.045+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
      debug 2024-11-14T08:14:21.045+0000 3ff97be5800  0 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-0/block failed: (22) Invalid argument
      debug 2024-11-14T08:14:21.045+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) open size 1099511627776 (0x10000000000, 1 TiB) block_size 4096 (4 KiB) non-rotational device, discard supported
      debug 2024-11-14T08:14:21.045+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) close
      debug 2024-11-14T08:14:21.245+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
      debug 2024-11-14T08:14:21.245+0000 3ff97be5800  0 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-0/block failed: (22) Invalid argument
      debug 2024-11-14T08:14:21.245+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) open size 1099511627776 (0x10000000000, 1 TiB) block_size 4096 (4 KiB) non-rotational device, discard supported
      debug 2024-11-14T08:14:21.245+0000 3ff97be5800  1 bdev(0x2aa35810e00 /var/lib/ceph/osd/ceph-0/block) close
      

      The OCP platform infrastructure and deployment type (AWS, Bare Metal, VMware, etc. Please clarify if it is platform agnostic deployment), (IPI/UPI):

       IBM Z - UPI

      The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):

       
      Internal Mode with LSO
       

      The version of all relevant components (OCP, ODF, RHCS, ACM whichever is applicable):

       
      OCP : 4.18.0-ec.3
      ODF : 4.18.0-49.stable
       

      Does this issue impact your ability to continue to work with the product?

       
      Yes
       

      Is there any workaround available to the best of your knowledge?

       
      No
       

      Can this issue be reproduced? If so, please provide the hit rate

       
      Yes
       

      Can this issue be reproduced from the UI?

      Yes

      If this is a regression, please provide more details to justify this:

      Steps to Reproduce:

      1.

      2.

      3.

      The exact date and time when the issue was observed, including timezone details:

       

      Actual results:

       

       

      Expected results:

       

      Logs collected and log location:

       

      Additional info:

       

              sostapov Scott Ostapovicz
              akandath Abdul Rauf Kuyyil Kandathil
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: