Description of problem:

As the number of lines in the output of a host's Remote Execution grows, the speed at which the WebUI displays results gets very poor. (using hammer, however, seems unaffected)

• • Someone double check the component I selected. I chose remote execution, but this is NOT a problem with the speed of actually executing. Rather it's just with viewing the output results once they're back on Satellite **

Version-Release number of selected component (if applicable):

Reported/reproduced on Satellite 6.8.6 through 6.9.3

How reproducible: Reliably

Steps to Reproduce:
1. (Presummption) - You can remote execute on a host
2. I created seven jobs to demonstrate this:

for i in

; do echo "Line ${i}: lest we get bored, here's a UUID: $(uuidgen)"; done
for i in

; do echo "Line ${i}: lest we get bored, here's a UUID: $(uuidgen)"; done
for i in

; do echo "Line ${i}: lest we get bored, here's a UUID: $(uuidgen)"; done
for i in

; do echo "Line ${i}: lest we get bored, here's a UUID: $(uuidgen)"; done
for i in

; do echo "Line ${i}: lest we get bored, here's a UUID: $(uuidgen)"; done
for i in

; do echo "Line ${i}: lest we get bored, here's a UUID: $(uuidgen)"; done
for i in

; do echo "Line ${i}: lest we get bored, here's a UUID: $(uuidgen)"; done

Under "Hosts"
-> "All Hosts"
-> Select a single host
-> "Schedule Remote Job"
-> I used "Run command - SSH default" and put the one of the above lines in each job.

3. Go to:
"Monitor" -> "Jobs".
For each of the above jobs, click the job description: https://satellite/job_invocations/<job_invocation_id>
Under the job invocation id, click the hostname to see the output: https://satellite/template_invocations/<template_invocation_id>

Actual results:

Viewing the output in the WebUI as described above results in a vastly skewed times, growing exponentially slower as output grows.

Number of lines in output | Wall clock time | Log reported Completed 200 OK in <n>ms
--------------------------------------------------------------------------------
10 | < 1 second | 85ms
100 | < 1 second | 281ms
1000 | 8-10 seconds | 2263ms
2000 | ~14 seconds | 4179ms
3000 | ~48 seconds | 4130ms
4000 | ~1min 43 seconds | 8141ms
5000 | ~3min 07 seconds | 9223ms

Expected results:

I would hope for something more in line with the hammer times, which were only a couple seconds in worst case.:

1. time hammer job-invocation output --id <job id) --host myhostname

10 lines : real 0m1.503s user 0m0.969s sys 0m0.119s
100 lines : real 0m1.491s user 0m0.966s sys 0m0.116s
1000 lines : real 0m1.640s user 0m1.032s sys 0m0.107s
2000 lines : real 0m1.752s user 0m1.088s sys 0m0.111s
3000 lines : real 0m1.752s user 0m1.088s sys 0m0.111s
4000 lines : real 0m2.100s user 0m1.212s sys 0m0.110s
5000 lines : real 0m2.238s user 0m1.204s sys 0m0.104s