LTC bug severity = Moderate

== Comment: #0 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2024-11-19 08:55:36 ==
Problem Summary:
=================
Memory mode mismatch between HMC and LPAR, When Memory mode in HMC is set to Dedicated Mode, but in LPAR it appears as Shared Mode. ?

Environment
Denali:
Kernel Build: 5.14.0-524.el9.ppc64le
Distro - Red Hat Enterprise Linux 9.6 Beta
Managed system: ltcden13
System Name :lp6
Platform : PowerVM
HMC : https://ltcvhmc5.onecloud.stglabs.ibm.com/(hscroot/lab-password)

----- uname -a -----
Linux localhost.localdomain 5.14.0-524.el9.ppc64le #1 SMP Wed Oct 30 10:14:26 EDT 2024 ppc64le ppc64le ppc64le GNU/Linux

----- lparstat -----
System Configuration
type=Shared mode=Uncapped smt=8 lcpu=1 mem=15061888 kB cpus=40 ent=0.05
%user %sys %wait %idle physc %entc lbusy app vcsw phint
----- ----- ----- ----- ----- ----- ----- ----- ----- -----
0.01 0.01 0.00 99.98 0.00 0.00 0.02 2670.66 3527827 86

----- lsmcode -----
Version of System Firmware is FW1060.00 (NH1060_031) (t) FW1060.00 (NH1060_031) (p) FW1060.00 (NH1060_031) (b)

— error logs ----
lparstat -i

Node Name : localhost.localdomain
Partition Name : ltcden13-lp6_naveen
Partition Number : 6
Type : Shared
Mode : Uncapped
Entitled Capacity : 0.05
Partition Group-ID : 0
Shared Pool ID : 0
Online Virtual CPUs : 1
Maximum Virtual CPUs : 1
Minimum Virtual CPUs : 1
Online Memory : 15061888 kB
Minimum Memory : 16384
Desired Memory : 16384
Maximum Memory : 18253611008
Minimum Capacity : 0.05
Maximum Capacity : 0.05
Capacity Increment : 0.01
Active Physical CPUs in system : 48
Active CPUs in Pool : 40
Shared Physical CPUS in system : 40
Maximum Capacity of Pool : 40.00
Entitled Capacity of Pool : 5
Unallocated Processor Capacity : 0
Physical CPU Percentage : 5
Unallocated Weight : 0
Memory Mode : Shared
Desired Variable Capacity Weight : 128

???----- Steps to reproduce -----
1. Verify that Memory Mode in HMC is Dedicated mode?2. Login to LPAR and execute commands lparstat -i and check Memory mode

----- Attachments -----

1. Screenshot of HMC
2. Installer Logs

== Comment: #1 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2024-11-19 09:34:37 ==

== Comment: #2 - Sachin Sant <ssant@in.ibm.com> - 2024-11-19 22:04:35 ==
The type and mode in lparstat o/p corresponds to processor. Please refer to lparstat man page.

Which setting has been used for processors and does it match with lparstat o/p?

== Comment: #3 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2024-11-20 03:11:31 ==
lparstat -i Lists details of LPAR Configuration.
Now from HMC Processor mode was set to Dedicated, capped and Memory mode was in Dedicated mode.
Output of lparstat -i shows Processor Dedicated mode, Capped type as expected but Memory mode(2nd row from bottom) shows Shared, That value was mismatching with HMC.

[root@ltcden13-lp6 ~]# lparstat -i
Node Name : ltcden13-lp6.aus.stglabs.ibm.com
Partition Name : ltcden13-lp6_naveen
Partition Number : 6
Type : Dedicated
Mode : Capped
Entitled Capacity : 1.00
Partition Group-ID : 0
Online Virtual CPUs : 1
Maximum Virtual CPUs : 1
Minimum Virtual CPUs : 1
Online Memory : 15061888 kB
Minimum Memory : 16384
Desired Memory : 16384
Maximum Memory : 16106127360
Minimum Capacity : 1.00
Maximum Capacity : 1.00
Capacity Increment : 1.00
Active Physical CPUs in system : 48
Active CPUs in Pool : 0
Shared Physical CPUS in system : 0
Maximum Capacity of Pool : 0.00
Entitled Capacity of Pool : 0
Unallocated Processor Capacity : 0
Physical CPU Percentage : 100
Unallocated Weight : 0
Memory Mode : Shared
Desired Variable Capacity Weight : 0

== Comment: #5 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2024-11-20 11:06:58 ==
[root@localhost ~]# cat /proc/ppc64/lparcfg
lparcfg 1.9
serial_number=IBM,02134C458
system_type=IBM,9080-HEX
partition_id=6
BoundThrds=1
CapInc=100
DisWheRotPer=5120000
MinEntCap=100
MinEntCapPerVP=100
MinMem=16384
MinProcs=1
partition_max_entitled_capacity=100
system_potential_processors=48
DesEntCap=100
DesMem=16384
DesProcs=1
DesVarCapWt=0
DedDonMode=0
CapiLicensed=1
ServicePartition=0
NumLpars=7
TimebaseOffset=0xFFFF40AD79000000

partition_entitled_capacity=100
group=0
system_active_processors=48
unallocated_capacity_weight=0
capacity_weight=0
capped=1
unallocated_capacity=0
physical_procs_allocated_to_virtualization=0
max_proc_capacity_available=0
entitled_proc_capacity_available=0
cmo_enabled=0
dispatches=1433413
dispatch_dispersions=0
purr=2356978665564
tbr=412080153653557
partition_affinity_score=100
partition_active_processors=1
partition_potential_processors=1
shared_processor_mode=0
power_mode_data=0002000000020002
MaxMem=16106127360
security_flavor=2

== Comment: #7 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2024-11-22 05:55:59 ==
Change Mode option was not available for Memory Mode in HMC or LPAR and apart from lparstat -i I couldn?t find any other command to verify Memory Mode .

== Comment: #8 - SEETEENA THOUFEEK <sthoufee@in.ibm.com> - 2024-12-03 00:14:30 ==
routing to HMC team to understand which variable in lparcfg make use when Memory mode in HMC is set to Shared Mode/dedicate mode.

== Comment: #9 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-03 00:16:50 ==
<===This is bridged from RTC description===>
This is the description of the defect added by LTC - EWM bridge automatically, bridged from LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=210654.
</===End of RTC description===>

== Comment: #10 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-03 00:53:09 ==
<PEDBG>NOT_FOUND</PEDBG>
<PEDBG_MESSAGE>
Please collect pedbg logs - run the command: pedbg -c -q 4 (as hscpe)
ssh dump@hmc-problem-old.aus.stglabs.ibm.com 'mkdir EWM_667000' (current lab password: https://stg-lab-password.dal1a.cirrus.ibm.com/)
scp <pedbg log file> dump@hmc-problem-old.aus.stglabs.ibm.com:~/EWM_667000/.
Notes:
1. If failure described in this defect, involes VIOS or CEC, then you may also collect snap and FFDC logs respectively
2. If CEC is involved, then please collect pedbg logs from all the HMCs to which CEC is connected.
</PEDBG_MESSAGE>

== Comment: #11 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-04 08:23:08 ==
Unable to login to HMC with any of the latest lab psaswords.
We will require pedbg logs as well.

If using GUI, it would be helpful to provide the screenshot. Else provide the command executed and its output on HMC.

== Comment: #12 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-04 14:43:13 ==
Please provide the information requested and reopen.

== Comment: #13 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2024-12-05 23:46:43 ==
Copied pedbg logs to specified path, so re-opening defect with pedbg logs.
please specify path for ffdc, snap and required screenshot in GUI so that I can share logs.

$ ls
HMClogs.ltcvhmc512060436.zip
Thu Dec 05 23:36:50
dump@hmc-problem-old:~/EWM_667000

== Comment: #14 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-06 00:33:07 ==
<PEDBG>FOUND</PEDBG>
<PEDBG_LOGFILES>
rw-r---- 1 dump dump 4772220156 Dec 5 23:30 /dumps//667000/HMClogs.ltcvhmc512060436.zip
</PEDBG_LOGFILES>

<AUTO_SCREEN_BASIC_INFO>
<PEDBG_LOG: /dumps//667000/HMClogs.ltcvhmc512060436.zip>
<HMC_NAME>ltcvhmc5</HMC_NAME>
<HMC_IP>9.40.206.255</HMC_IP>
<HMC_LEVEL>
"version= Version: 10
Release: 3
Service Pack: 1060
HMC Build level 2404100258
UNASSIGNED - PTF mapping is unassigned. Please specify in src/data/ptfMapping.json
","base_version=V10R3
"
</HMC_LEVEL>
<CEC_INFO>
<LSSYSCFG>
name=ltcden13,type_model=9080-HEX,serial_num=134C458,ipaddr=9.53.174.179,ipaddr_secondary=9.53.174.180,sp_type=fsp,state=Operating,state_detail=
</LSSYSCFG>
<LSSYSCONN>
resource_type=sys,type_model_serial_num=9080-HEX*134C458,sp_type=fsp,sp=primary,sp_phys_loc=U78D6.SC1.KIC3793-P1-C3,ipaddr=9.53.174.179,alt_ipaddr=unavailable,state=Connected
</LSSYSCONN>
<LSLIC>
lic_type=Managed System,management_status=Enabled,disabled_reason=,activated_level=31,activated_spname=FW1060.00,installed_level=31,installed_spname=FW1060.00,accepted_level=31,accepted_spname=FW1060.00,ecnumber=01NH1060,mtms=9080-HEX*134C458,deferred_level=None,deferred_spname=None,platform_ipl_level=31,platform_ipl_spname=FW1060.00,curr_level_primary=31,curr_spname_primary=FW1060.00,curr_ecnumber_primary=01NH1060,curr_power_on_side_primary=perm,pend_power_on_side_primary=perm,temp_level_primary=31,temp_spname_primary=FW1060.00,temp_ecnumber_primary=01NH1060,perm_level_primary=31,perm_spname_primary=FW1060.00,perm_ecnumber_primary=01NH1060,update_control_primary=HMC,update_access_key_exp_date=11/18/2022,curr_level_secondary=31,curr_spname_secondary=FW1060.00,curr_ecnumber_secondary=01NH1060,curr_power_on_side_secondary=perm,pend_power_on_side_secondary=perm,temp_level_secondary=31,temp_spname_secondary=FW1060.00,temp_ecnumber_secondary=01NH1060,perm_level_secondary=31,perm_spname_secondary=FW1060.00,perm_ecnumber_secondary=01NH1060,update_control_secondary=HMC
</LSLIC>
<GET_CEC_TIME_OF_DAY>
2807:2024-12-05 11:28:26.870 [Thread-503--158604] :: calling doFSPCommand: GET_CEC_TIME_OF_DAY vport=7 tg=11 op=402 ip=9.53.174.179
2835:2024-12-05 11:28:26.876 [Thread-503--158604] :: cecTime: year=2024; month=12; day=5; hour=11; minute=27; second=36; microSec=655515;1733398057531
</GET_CEC_TIME_OF_DAY>
</CEC_INFO>
</PEDBG_LOG>
</AUTO_SCREEN_BASIC_INFO>

== Comment: #15 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-06 06:23:08 ==
Added: attachment 37610 (Screenshot 2024-12-06 at 5.46.23 PM.png)

== Comment: #16 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-06 06:33:07 ==
Attaching the screenshot shared by NAveen

== Comment: #17 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-08 23:43:07 ==
The UI shows the Memory as dedicated which is as expected. The issue is with below command returning incorrect memory mode value.
Transferring to provider team for investigation.

[root@ltcden13-lp6 ~]# lparstat -i
Node Name : ltcden13-lp6.aus.stglabs.ibm.com
Partition Name : ltcden13-lp6_naveen
Partition Number : 6
Type : Dedicated
Mode : Capped
Entitled Capacity : 1.00
Partition Group-ID : 0
Online Virtual CPUs : 1
Maximum Virtual CPUs : 1
Minimum Virtual CPUs : 1
Online Memory : 15061888 kB
Minimum Memory : 16384
Desired Memory : 16384
Maximum Memory : 16106127360
Minimum Capacity : 1.00
Maximum Capacity : 1.00
Capacity Increment : 1.00
Active Physical CPUs in system : 48
Active CPUs in Pool : 0
Shared Physical CPUS in system : 0
Maximum Capacity of Pool : 0.00
Entitled Capacity of Pool : 0
Unallocated Processor Capacity : 0
Physical CPU Percentage : 100
Unallocated Weight : 0
Memory Mode : Shared
Desired Variable Capacity Weight : 0

== Comment: #18 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-10 08:23:09 ==
On HMC, I see partition is in Dedicated Mode, not sure why we are seeing Shared on OS.

[root@ltcvhmc5 ~] # lshwres -m ltcden13 -r mem --level lpar --filter "lpar_names=ltcden13-lp6_naveen"
lpar_name=ltcden13-lp6_naveen,lpar_id=6,curr_min_mem=16384,curr_mem=16384,curr_max_mem=16384,pend_min_mem=16384,pend_mem=16384,pend_max_mem=16384,run_min_mem=16384,run_mem=16384,curr_min_num_huge_pages=0,curr_num_huge_pages=0,curr_max_num_huge_pages=0,pend_min_num_huge_pages=0,pend_num_huge_pages=0,pend_max_num_huge_pages=0,run_num_huge_pages=0,mem_mode=ded,curr_mem_expansion=0.0,pend_mem_expansion=0.0,desired_hardware_mem_expansion=1,curr_hardware_mem_expansion=1,desired_hardware_mem_encryption=1,curr_hardware_mem_encryption=1,curr_hpt_ratio=1:128,curr_ppt_ratio=1:4096,kvm_reserved_mem_percent=0
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lparstat is not HMC command, it a OS command. Need OS team to check why the Memory Mode is displayed as Shared.

== Comment: #21 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-14 13:23:16 ==
[dump@hmc-problem-old dumps]$ mv -i 667000 EWM_667000

== Comment: #22 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2024-12-14 14:03:08 ==
[dump@hmc-problem-old ~]$ su -
Password:
Last login: Sat Dec 14 13:58:48 CST 2024 on pts/1
-bash-4.2# rmdir /home/dump/EWM_667000

== Comment: #26 - Shrikanth Hegde <Shrikanth.Hegde@ibm.com> - 2025-01-09 03:23:47 ==
Hi.

Patch has been posted upstream. Please test it out.
https://groups.google.com/g/powerpc-utils-devel/c/ASm9ihz_XPY

Thanks
Shrikanth

== Comment: #27 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2025-01-15 22:55:56 ==
Verified in System, now I am able to see Memory Mode as Dedicated
[root@ltcden13-lp6 src]# ./lparstat -i
Node Name : ltcden13-lp6.aus.stglabs.ibm.com
Partition Name : ltcden13-lp6_naveen
Partition Number : 6
Type : Dedicated
Mode : Capped
Entitled Capacity : 10.00
Partition Group-ID : 0
Online Virtual CPUs : 10
Maximum Virtual CPUs : 15
Minimum Virtual CPUs : 2
Online Memory : 15548992 kB
Minimum Memory : 16384
Desired Memory : 16384
Maximum Memory : 20937965568
Minimum Capacity : 2.00
Maximum Capacity : 15.00
Capacity Increment : 1.00
Active Physical CPUs in system : 48
Active CPUs in Pool : 0
Shared Physical CPUS in system : 0
Maximum Capacity of Pool : 0.00
Entitled Capacity of Pool : 0
Unallocated Processor Capacity : 0
Physical CPU Percentage : 100
Unallocated Weight : 0
Memory Mode : Dedicated
Desired Variable Capacity Weight : 0

== Comment: #28 - Thummalapenta, Naveen Kumar <Naveen.Kumar.Thummalapenta@ibm.com> - 2025-01-16 02:53:51 ==
Verified in System, now I am able to see Memory Mode as Dedicated
[root@ltcden13-lp6 src]# ./lparstat -i
Node Name : ltcden13-lp6.aus.stglabs.ibm.com
Partition Name : ltcden13-lp6_naveen
Partition Number : 6
Type : Dedicated
Mode : Capped
Entitled Capacity : 10.00
Partition Group-ID : 0
Online Virtual CPUs : 10
Maximum Virtual CPUs : 15
Minimum Virtual CPUs : 2
Online Memory : 15548992 kB
Minimum Memory : 16384
Desired Memory : 16384
Maximum Memory : 20937965568
Minimum Capacity : 2.00
Maximum Capacity : 15.00
Capacity Increment : 1.00
Active Physical CPUs in system : 48
Active CPUs in Pool : 0
Shared Physical CPUS in system : 0
Maximum Capacity of Pool : 0.00
Entitled Capacity of Pool : 0
Unallocated Processor Capacity : 0
Physical CPU Percentage : 100
Unallocated Weight : 0
Memory Mode : Dedicated
Desired Variable Capacity Weight : 0

please integrate the patch once accepted upstream.