Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-7744

Fix moving / banning clone and bundle resources

XMLWordPrintable

    • pcs-0.11.6-6.el9
    • Normal
    • 25e28913ae7ac07cdfa8c39c3c4920b4823839e7 25a5a4bb4b1499b425707c5e736bd183ee97ee8d 3a456616f5e9453e59d7b6f712444692d6a4ad07 c2c7c033aaff8752587959946441fde84e10dff9 a74e35a30c35e00cbcc680cac4241ff8922db086 10e637e96f01ed78667a829344ddacd098b47a6d
    • sst_high_availability
    • ssg_filesystems_storage_and_HA
    • 13
    • 19
    • 5
    • QE ack
    • False
    • Hide

      None

      Show
      None
    • Yes
    • Bug Fix
    • Hide
      .Issues with moving and banning clone and bundle resources now corrected

      This bug fix addresses two limitations of moving bundled and clone resources:

      * When a user tried to move a bundled resource out of its bundle or ban it from running in its bundle, `pcs` created a constraint but the constraint had no effect. This caused the move to fail with an error message. With this fix, `pcs` disallows moving and banning bundled resources from their bundles and prints an error message noting that bundled resources cannot be moved out of their bundles.

      * When a user tried to move a bundle or clone resource, `pcs` exited with an error message noting that bundle or clone resources cannot be moved. This fix relaxes validation of move commands. It is now possible to move clone and bundle resources. When moving clone resources, you must specify a destination node if more than one instance of a clone is running. Only one-replica bundles can be moved.
      Show
      .Issues with moving and banning clone and bundle resources now corrected This bug fix addresses two limitations of moving bundled and clone resources: * When a user tried to move a bundled resource out of its bundle or ban it from running in its bundle, `pcs` created a constraint but the constraint had no effect. This caused the move to fail with an error message. With this fix, `pcs` disallows moving and banning bundled resources from their bundles and prints an error message noting that bundled resources cannot be moved out of their bundles. * When a user tried to move a bundle or clone resource, `pcs` exited with an error message noting that bundle or clone resources cannot be moved. This fix relaxes validation of move commands. It is now possible to move clone and bundle resources. When moving clone resources, you must specify a destination node if more than one instance of a clone is running. Only one-replica bundles can be moved.
    • Done

      +++ This bug was initially created as a clone of Bug #1578789 +++

      Description of problem:

      • moving and banning an inner bundle resource
        • pcs should exit with an error saying moving / banning inner bundle resources is not allowed
      • moving / banning a pacemaker internal bundle resource
        • works OK, pcs says the resource doesn't exist (it's not in the CIB)
      • moving and banning a bundle resource
        • does not currently work in crm_resource even for 1-instance bundles:
          Resource 'DummyBundle' not moved: active in 2 locations.
      • moving and banning a clone resource
        • works in crm_resource for 1-instance clones for both a clone and its inner resource
        • for multi-instance clones, moving a resource in a clone creates a constraint, whereas moving a clone resource exits with an error
        • pcs should not allow moving / banning a resource in a clone, then it can simply run crm_resource every time and based on its output inform a user if a clone could or couldn't be moved / banned
        • another option is to automatically switch to moving / banning a bundle / clone if the user tries to move / ban a resource in the bundle / clone

      Version-Release number of selected component (if applicable):

      pcs-0.9.162-5.el7.x86_64

      How reproducible:

      always, easily

      Steps to Reproduce:

      Performing tests of move without node specification

      dummy0 is a resource inside DummyBundle bundle with replicas=1

       

      [root@virt-143 ~]# pcs resource move dummy0
Warning: Creating location constraint cli-ban-dummy0-on-DummyBundle-0 with a score of -INFINITY for resource dummy0 on node DummyBundle-0.
This will prevent dummy0 from running on DummyBundle-0 until the constraint is removed. This will be the case even if DummyBundle-0 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy0
Disabled on: DummyBundle-0 (score:-INFINITY) (role: Started) (id:cli-ban-dummy0-on-DummyBundle-0)
[root@virt-143 ~]# crm_resource --resource dummy0 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy0-on-DummyBundle-0' with a score of -INFINITY for resource dummy0 on DummyBundle-0.
This will prevent dummy0 from running on DummyBundle-0 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if DummyBundle-0 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy0
Disabled on: DummyBundle-0 (score:-INFINITY) (role: Started) (id:cli-ban-dummy0-on-DummyBundle-0)

       

      Constraint has been created, but resource WAS NOT moved. The reason is obvious as DummyBundle-0 is not an actual node. This is a possible place for improvement.

       

      [root@virt-143 ~]# pcs resource move DummyBundle-0
Error: DummyBundle-0 is not a valid resource
[root@virt-143 ~]# crm_resource --resource DummyBundle-0 --move
WARNING: Creating rsc_location constraint 'cli-ban-DummyBundle-0-on-virt-143' with a score of -INFINITY for resource DummyBundle-0 on virt-143.
This will prevent DummyBundle-0 from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-143 is the last node in the cluster
This message can be disabled with --quiet
Error performing operation: Update does not conform to the configured schema

       

      Neither pcs not crm_resource is able to target the DummyBundle-0 pseudo resource which is expected.

       

      [root@virt-143 ~]# pcs resource move DummyBundle
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource DummyBundle --move
Resource 'DummyBundle' not moved: active in 2 locations.
You can prevent 'DummyBundle' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

       

      In this situation pcs simply shouts for trying to move a clone, while crm_resource gives us something else - a misinformation about the number of locations being active. There should only be one active due to replicas=1. Let's see how well the pure clones will do:

      dummy1 is a clone with clone-max=1

       

      [root@virt-143 ~]# pcs resource move dummy1
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource dummy1 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-145' with a score of -INFINITY for resource dummy1 on virt-145.
This will prevent dummy1 from running on virt-145 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-145 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Disabled on: virt-145 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-145)

       

      A clone with single instance was refused to move by pcs, but actually passed and WAS moved with crm_resource. There might be a room for improvement in pcs.

       

      [root@virt-143 ~]# pcs resource move dummy1-clone
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource dummy1-clone --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-clone-on-virt-143' with a score of -INFINITY for resource dummy1-clone on virt-143.
This will prevent dummy1-clone from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-143 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1-clone
Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-clone-on-virt-143)

       

      When targeting clone ID, once again a clone with single instance was refused to move by pcs, but passed and WAS moved with crm_resource.

      dummy1 is a clone with clone-max=2

       

      [root@virt-143 ~]# pcs resource move dummy1
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource dummy1 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-143' with a score of -INFINITY for resource dummy1 on virt-143.
This will prevent dummy1 from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-143 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)
[root@virt-143 ~]# crm_resource --resource dummy1 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-145' with a score of -INFINITY for resource dummy1 on virt-145.
This will prevent dummy1 from running on virt-145 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-145 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)
Disabled on: virt-145 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-145)

       

      With more than one clone instance crm_resource does surprisingly perform a constraint-based move by moving ONE of the instance. Even if ran twice in a row.

       

      [root@virt-143 ~]# pcs resource move dummy1-clone
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource dummy1-clone --move
Resource 'dummy1-clone' not moved: active in 2 locations.
You can prevent 'dummy1-clone' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

       

      Here with clone ID and two instances crm_resource won't do us a favor anymore. Pcs keeps printing the same error as before.

      So, the conclusion for 'resource move':

      • pcs is not really up in sync with crm_resource behavior in terms of being able to move a single clone instance
      • support for moving a bundle in crm_resource is not on par with being able to move a clone, at least in case of just one instance being present due to:
      • 2 active locations being improperly reported
      • the move constraint of a primitive is incorrectly targeted to a non-existent DummyBundle-0 node.

      It is worth pointing out that creating a pure location constraint for DummyBundle will move the only existing bundle instance to the desired node as one would expect. I yet have to test the behavior of 'resource ban'.

      Performing tests of move with node specification

      dummy0 is a resource inside DummyBundle bundle with replicas=1

       

      [root@virt-143 ~]# pcs resource move dummy0 virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy0
Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy0)
[root@virt-143 ~]# crm_resource --resource dummy0 --move --node virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy0
Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy0)

       

      Constraint has been created, but resource WAS NOT moved.

       

      [root@virt-143 ~]# pcs resource move DummyBundle-0 virt-145
Error: DummyBundle-0 is not a valid resource
[root@virt-143 ~]# crm_resource --resource DummyBundle-0 --move --node virt-145
Error performing operation: Update does not conform to the configured schema

       

      Neither pcs not crm_resource is able to target the DummyBundle-0 pseudo resource which is expected.

       

      [root@virt-143 ~]# pcs resource move DummyBundle virt-145
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource DummyBundle --move --node virt-145
Resource 'DummyBundle' not moved: active on multiple nodes
Error performing operation: Invalid argument

       

      Just like in the previous case the crm_resource gives us a misinformation about the number of locations being active. There should only be one active due to replicas=1.

      dummy1 is a clone with clone-max=1

       

      [root@virt-143 ~]# pcs resource move dummy1 virt-145
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource dummy1 --move --node virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1)

       

      A clone with single instance was refused to move by pcs, but actually passed and WAS moved with crm_resource. There might be a room for improvement in pcs.

       

      [root@virt-143 ~]# pcs resource move dummy1-clone virt-146
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource dummy1-clone --move --node virt-146
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1-clone
Enabled on: virt-146 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1-clone)

       

      When targeting clone ID, once again a clone with single instance was refused to move by pcs, but passed and WAS moved with crm_resource.

      dummy1 is a clone with clone-max=2

       

      [root@virt-143 ~]# pcs resource move dummy1 virt-145
Error: cannot move cloned resources
[root@virt-143 ~]# crm_resource --resource dummy1 --move --node virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1)

       

      With more than one clone instance crm_resource does perform a constraint-based move by moving ONE of the instance.

      [root@virt-143 ~]# pcs resource move dummy1-clone virt-146
Error: cannot move cloned resources
[root@virt-143 ~]# 
[root@virt-143 ~]# crm_resource --resource dummy1-clone --move --node virt-146
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1-clone
Enabled on: virt-146 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1-clone)

      Here with clone ID and two instances crm_resource won't do us a favor anymore. Pcs keeps printing the same error as before.

      The conclusion for 'resource move' with node specification is pretty much aligned with the previous one:

      • pcs is not really up in sync with crm_resource behavior in terms of being able to move a single clone instance
      • support for moving a bundle in crm_resource is not on par with being able to move a clone, at least in case of just one instance being present due to:
      • 2 active locations being improperly reported
      • the move constraint of a primitive is incorrectly targeted to a non-existent DummyBundle-0 node.

      Performing tests of ban with crm_resource is pretty much identical to simple move (with no node specification), so I am going to mention pcs differences only:

      dummy0 is a resource inside DummyBundle bundle with replicas=1

       

      [root@virt-143 ~]# pcs resource ban DummyBundle
Error: error moving/banning/clearing resource
Resource 'DummyBundle' not moved: active in 2 locations.
You can prevent 'DummyBundle' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument
[root@virt-143 ~]# crm_resource --resource DummyBundle --ban
Resource 'DummyBundle' not moved: active in 2 locations.
You can prevent 'DummyBundle' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

       

      The difference is in pcs output when targetting DummyBundle which no longer complains about a clone but instead passes the crm_resource message as is. We'll need a more user friendly message in pcs.

      dummy1 is a clone with clone-max=1

       

      [root@virt-143 ~]# pcs resource ban dummy1
Warning: Creating location constraint cli-ban-dummy1-on-virt-143 with a score of -INFINITY for resource dummy1 on node virt-143.
This will prevent dummy1 from running on virt-143 until the constraint is removed. This will be the case even if virt-143 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)
[root@virt-143 ~]# crm_resource --resource dummy1 --ban
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-144' with a score of -INFINITY for resource dummy1 on virt-144.
This will prevent dummy1 from running on virt-144 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-144 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Disabled on: virt-144 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-144)

       

      The resource WAS moved in both cases so pcs and crm_resource are in sync regarding behavior.

       

      [root@virt-143 ~]# pcs resource ban dummy1-clone
Warning: Creating location constraint cli-ban-dummy1-clone-on-virt-143 with a score of -INFINITY for resource dummy1-clone on node virt-143.
This will prevent dummy1-clone from running on virt-143 until the constraint is removed. This will be the case even if virt-143 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1-clone
Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-clone-on-virt-143)
[root@virt-143 ~]# crm_resource --resource dummy1-clone --ban
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-clone-on-virt-144' with a score of -INFINITY for resource dummy1-clone on virt-144.
This will prevent dummy1-clone from running on virt-144 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-144 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1-clone
Disabled on: virt-144 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-clone-on-virt-144)

       

      Again the resource WAS moved in both cases so pcs and crm_resource are in sync regarding behavior.

      dummy1 is a clone with clone-max=2

       

      [root@virt-143 ~]# pcs resource ban dummy1
Warning: Creating location constraint cli-ban-dummy1-on-virt-145 with a score of -INFINITY for resource dummy1 on node virt-145.
This will prevent dummy1 from running on virt-145 until the constraint is removed. This will be the case even if virt-145 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Disabled on: virt-145 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-145)
[root@virt-143 ~]# crm_resource --resource dummy1 --ban
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-143' with a score of -INFINITY for resource dummy1 on virt-143.
This will prevent dummy1 from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
This will be the case even if virt-143 is the last node in the cluster
This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
Resource: dummy1
Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)

      One of the resource instances WAS moved in both cases so pcs and crm_resource are in sync regarding behavior.

       

      [root@virt-143 ~]# pcs resource ban dummy1-clone
Error: error moving/banning/clearing resource
Resource 'dummy1-clone' not moved: active in 2 locations.
You can prevent 'dummy1-clone' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument
[root@virt-143 ~]# 
[root@virt-143 ~]# crm_resource --resource dummy1-clone --ban
Resource 'dummy1-clone' not moved: active in 2 locations.
You can prevent 'dummy1-clone' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

       

      The difference is in pcs output when targetting DummyBundle which no longer complains about a clone but instead passes the crm_resource message as is. We'll need a more user friendly message in pcs.

      The conclusion for 'resource ban' is the same for crm_resource, things are only different for pcs:

      • pcs now doesn't scream about clones anymore but instead pass crm_resource's error message directly, which we might need to modify a little
      • support for banning a bundle in crm_resource is not on par with being able to move a clone, at least in case of just one instance being present due to:
      • 2 active locations being improperly reported
      • the move constraint of a primitive is incorrectly targeted to a non-existent DummyBundle-0 node

      — Additional comment from Tomas Jelinek on 2023-07-25 15:23:54 CEST —

      Summary after bz1578820 has been resolved:

      'pcs resource ban' works as described in comment 0. Such behavior is fine and there is nothing to be fixed. Pcs delegates the banning to crm_resource, including detection of number of location a resource is active in.

      'pcs resource move-with constraint' and 'pcs resource move' needs to be fixed:

      • Moving / banning an inner bundle resource from its bundle should be disallowed in pcs. Currently, pcs doesn't check for this and creates a constraint. The constraint has no effect, as expected.
      • Moving a 1-replica bundle works, but pcs does not allow it. There is a check in a validator in 'pcs resource move' command which prevents moving all bundle resources. As moving a 1-replica bundle works in pacemaker, and moving multi-replica bundles is prevented by pacemaker, the check in pcs can be removed completely.
      • Moving a clone resource works the same as moving a bundle resource and is disallowed by pcs in the same way as well. Removing this limitation from pcs should be considered.

            cluster-qe Cluster QE
            tojeline@redhat.com Tomas Jelinek
            Tomas Jelinek Tomas Jelinek
            Nina Hostakova Nina Hostakova
            Steven Levine Steven Levine
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: