Description of problem:

When read affinity is enabled, we are observing drop in IO performance for sequential reads with 256KB block size compared to the performance when read affinity is disabled.
We executed few more tests with increased OSD CPU and Memory from the default values. In all cases, the observation remains the same.

---------------------------------------------------------------------------------
Sequential Reads, Throughput (GB/s) |
---------------------------------------------------------------------------------
READ_AFFINITY | Default | Increased OSD Mem | Increased OSD CPU |

---------------------------------------------------------------------------------
TRUE | 3.999 | 4.117 | 4.433 |
---------------------------------------------------------------------------------
FALSE | 4.275 | 4.636 | 5.479 |
---------------------------------------------------------------------------------
% change | -6.46% | -11.19% | -19.09% |
---------------------------------------------------------------------------------

On closer inspection, it was observed that OSD memory is probably the bottleneck which is not the case when read affinity is disabled.
The test results and performance charts are mentioned in this doc: https://docs.google.com/document/d/17TkI0kqg2i7yX9WI5WToaVC6Vn8EYNJ0sAw5kTqj8HM/edit?usp=sharing

No such behaviour was observed for random reads with smaller block sizes (4KB, 8KB), instead we are getting significant improved performance which is expected when read affinity is enabled.

Is this an expected behaviour for sequential reads with large block sizes when read affinity is enabled?

Version of all relevant components (if applicable): ODF 4.13

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)? No

Is there any workaround available to the best of your knowledge? No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)? 1

Can this issue reproducible? - Yes

Can this issue reproduce from the UI? - No

If this is a regression, please provide more details to justify this: NA

Steps to Reproduce:
1. Install ODF in a baremetal OCP cluster.
2. Enable read affinity.
3. Execute sequential reads with 256KB block size.

Actual results: The performance of sequential reads are lower when read affity is enabled when compared to the performance when read affinity is disabled.

Expected results: The performance should improve or at least remain the same when read affinity is enabled.

Additional info: None