Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-70165

LVMCluster does not become ready

XMLWordPrintable

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      This is an LVMS Bug Report:

      Please create & attach a must-gather as indicated by this Guide to collect LVMS relevant data from the Cluster (linked to the latest version, use older versions of the documentation for older OCP releases as applicable

      Please make sure that you describe your storage configuration in detail. List all devices that you plan to work with for LVMS as well as any relevant machine configuration data to make it easier for an engineer to help out.

      Description of problem:

      Did a re-installation of LVMS due to some strange issues. Followed the clean/force uninstallation and then proceeded with a fresh installation.
      Config is to use local SSDs on 4 nodes out of a >10 node cluster. Using optionalPath to the disk-by-id path on the nodes, plus a node selected downselecting to the 4 relevant nodes. (see LVMCluster object)
      
      LVMCluster object is stuck in state "Progressing" with message
       DaemonSet is not considered ready: the DaemonSet is not ready: openshift-lvm-storage/vg-manager. 2 out of 3 expected pods are ready' 
      
      Which is already strange, because the DaemonSet actullay has 4 (not 3) pods:
       % oc get DaemonSet
      NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
      vg-manager   4         4         2       4            2           <none>          7m16s
      
      Logs of a "not ready"pod: show multiple errors:
      
      % oc logs -f vg-manager-l278q 
      {"level":"info","ts":"2025-12-23T12:00:24Z","msg":"Waiting for lock","lockFile":"/var/lock/vgmanager/vgmanager.lock"}
      {"level":"info","ts":"2025-12-23T12:00:24Z","msg":"Lock acquired","lockFile":"/var/lock/vgmanager/vgmanager.lock"}
      {"level":"error","ts":"2025-12-23T12:00:24Z","logger":"setup","msg":"lvmd config could not be loaded, starting without topolvm components and attempting bootstrap","error":"open /etc/topolvm/lvmd.yaml: no such file or directory","stacktrace":"github.com/openshift/lvm-operator/v4/cmd/vgmanager.run\n\t/workspace/cmd/vgmanager/vgmanager.go:187\ngithub.com/openshift/lvm-operator/v4/cmd/vgmanager.runWithFileLock\n\t/workspace/cmd/vgmanager/vgmanager.go:128\ngithub.com/openshift/lvm-operator/v4/cmd/vgmanager.NewCmd.func1\n\t/workspace/cmd/vgmanager/vgmanager.go:97\ngithub.com/spf13/cobra.(*Command).execute\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1071\nmain.main\n\t/workspace/cmd/main.go:33\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:283"}
      {"level":"info","ts":"2025-12-23T12:00:24Z","logger":"setup","msg":"starting manager"}
      ...
      ...
      {"level":"error","ts":"2025-12-23T12:00:24Z","msg":"Reconciler error","controller":"lvmvolumegroup","controllerGroup":"lvm.topolvm.io","controllerKind":"LVMVolumeGroup","LVMVolumeGroup":{"name":"local-ssd","namespace":"openshift-lvm-storage"},"namespace":"openshift-lvm-storage","name":"local-ssd","reconcileID":"9d062dba-b558-4f36-aafd-fa856c37b707","error":"could not get LVMVolumeGroupNodeStatus: LVMVolumeGroupNodeStatus.lvm.topolvm.io \"storm2\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}

       

      Version-Release number of selected component (if applicable):

       4.20

      Additional info:

      Cluster is available on Red Hat VPN - ping me on slack for access 
      MustGather will be uploaded to gdrive soon

              bzamalut@redhat.com Bulat Zamalutdinov
              dfroehli42rh Daniel Fröhlich
              None
              None
              Minal Pradeep Makwana Minal Pradeep Makwana
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: