Description of problem:
Seemingly harmless warning in pacemaker.log after FFU, related to resources that were present in 16.2 but no longer managed by pacemaker in 17.1

Version-Release number of selected component (if applicable):
17.1

How reproducible:
Always reproducible

Steps to Reproduce:
1. Deploy RHOSP 16.2
2. Perform FFU to 17.1 according to steps in the documentation

Actual results:
3. After the overcloud upgrade on the controllers [0], the following warnings show up on /var/log/pacemaker/pacemaker.log:

~~~
[root@overcloud-controller-0 ~]# grep "Ignoring recorded node state" /var/log/pacemaker/pacemaker.log | tail -6
Nov 20 06:57:42.116 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'redis-bundle-0' because it is no longer in the configuration
Nov 20 06:57:42.116 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'redis-bundle-1' because it is no longer in the configuration
Nov 20 06:57:42.116 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'redis-bundle-2' because it is no longer in the configuration
Nov 20 06:57:42.116 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'ovn-dbs-bundle-0' because it is no longer in the configuration
Nov 20 06:57:42.116 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'ovn-dbs-bundle-1' because it is no longer in the configuration
Nov 20 06:57:42.116 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'ovn-dbs-bundle-2' because it is no longer in the configuration
[root@overcloud-controller-0 ~]#
[root@overcloud-controller-0 ~]# grep -c "Ignoring recorded node state" /var/log/pacemaker/pacemaker.log
342
[root@overcloud-controller-0 ~]#
~~~

These warnings are coming up every 15 minutes, despite reboots.

Expected results:
No warnings for services that were removed during FFU should be on the log.

Additional info:
The running cib tree shows the problematic elements:
~~~
[root@overcloud-controller-0 ~]# cibadmin --query --local | egrep "redis-bundle|ovn-dbs-bundle"
<node_state remote_node="true" id="redis-bundle-0" uname="redis-bundle-0" in_ccm="false" crm-debug-origin="do_state_transition" node_fenced="0">
<lrm id="redis-bundle-0">
<node_state remote_node="true" id="redis-bundle-1" uname="redis-bundle-1" in_ccm="false" crm-debug-origin="do_state_transition" node_fenced="0">
<lrm id="redis-bundle-1">
<node_state remote_node="true" id="redis-bundle-2" uname="redis-bundle-2" in_ccm="false" crm-debug-origin="do_state_transition" node_fenced="0">
<lrm id="redis-bundle-2">
<node_state remote_node="true" id="ovn-dbs-bundle-0" uname="ovn-dbs-bundle-0" in_ccm="false" crm-debug-origin="do_state_transition" node_fenced="0">
<lrm id="ovn-dbs-bundle-0">
<node_state remote_node="true" id="ovn-dbs-bundle-1" uname="ovn-dbs-bundle-1" in_ccm="false" crm-debug-origin="do_state_transition" node_fenced="0">
<lrm id="ovn-dbs-bundle-1">
<node_state remote_node="true" id="ovn-dbs-bundle-2" uname="ovn-dbs-bundle-2" in_ccm="false" crm-debug-origin="do_state_transition" node_fenced="0">
<lrm id="ovn-dbs-bundle-2">
[root@overcloud-controller-0 ~]#
~~~

For the moment, I'm working this around by manually removing the elements based on the information above:
~~~
[root@overcloud-controller-0 ~]# cibadmin --delete --xml-text '<node_state id="redis-bundle-0"/>'
[root@overcloud-controller-0 ~]# cibadmin --delete --xml-text '<node_state id="redis-bundle-1"/>'
[root@overcloud-controller-0 ~]# cibadmin --delete --xml-text '<node_state id="redis-bundle-2"/>'
[root@overcloud-controller-0 ~]# cibadmin --delete --xml-text '<node_state id="ovn-dbs-bundle-0"/>'
[root@overcloud-controller-0 ~]# cibadmin --delete --xml-text '<node_state id="ovn-dbs-bundle-1"/>'
[root@overcloud-controller-0 ~]# cibadmin --delete --xml-text '<node_state id="ovn-dbs-bundle-2"/>'
[root@overcloud-controller-0 ~]# date
Wed Nov 20 07:09:50 AM UTC 2024
~~~

Then no more warnings come up on the log afterwards:
~~~
[root@overcloud-controller-0 ~]# date ; grep "Ignoring recorded node state" /var/log/pacemaker/pacemaker.log | tail -3
Wed Nov 20 07:41:57 AM UTC 2024
Nov 20 07:09:43.518 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'ovn-dbs-bundle-1' because it is no longer in the configuration
Nov 20 07:09:43.518 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'ovn-dbs-bundle-2' because it is no longer in the configuration
Nov 20 07:09:45.977 overcloud-controller-0 pacemaker-schedulerd[2374] (unpack_node_state) warning: Ignoring recorded node state for 'ovn-dbs-bundle-2' because it is no longer in the configuration
[root@overcloud-controller-0 ~]#
~~~

I would like to get confirmation that this is a valid fix/workaround before passing it to the customer. I can provide reproducer if needed.

[0] https://docs.redhat.com/en/documentation/red_hat_openstack_platform/17.1/html/framework_for_upgrades_16.2_to_17.1/upgrading-a-standard-overcloud_upgrading-overcloud-standard#upgrading-a-standard-overcloud_upgrading-overcloud-standard