+++ This bug was initially created as a clone of Bug #2266197 +++

Description of problem:

This happens after the 6.14 upgrade and it seems to match the upstream issue:

https://community.theforeman.org/t/content-view-list-takes-too-long/36495

— Additional comment from on 2024-02-27T00:23:23Z

The PR https://github.com/Katello/katello/pull/10865 does not seem solving the problem:

https://community.theforeman.org/t/content-view-list-takes-too-long/36495/47

I'm asking the customer to share a backup if possible to set up a reproducer.

— Additional comment from on 2024-02-27T06:51:39Z

The customer updated with a new finding:

~~~
As soon as I delete all tasks (foreman-rake foreman_tasks:cleanup TASK_SEARCH='label ~ *' STATES='running,pending,paused,stopped' VERBOSE=true) performance is normal for a short time.
~~~

however, I don't have time to set up a reproducer today.

— Additional comment from on 2024-02-28T06:45:58Z

Reproducer:

https://10.6.4.201/content_views

admin/redhat
root/redhat

— Additional comment from on 2024-02-28T06:51:25Z

2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters:

{"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}
[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters: {"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}

[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
The slowness is from this request:

[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters:

{"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}
[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters: {"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}

[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters:

{"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}
[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters: {"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}

[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters:

{"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}
[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters: {"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}

[root@dell-per6515-01 satellite-backup-2024-02-27-06-24-22]# grep a77bc44e /var/log/foreman/production.log
2024-02-28T01:49:50 [I|app|a77bc44e] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 01:49:50 -0500
2024-02-28T01:49:50 [I|app|a77bc44e] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T01:49:50 [I|app|a77bc44e] Parameters:

{"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}
2024-02-28T01:50:46 [I|app|a77bc44e] Rendered /usr/share/gems/gems/katello-4.9.0.21/app/views/katello/api/v2/content_views/index.json.rabl within katello/api/v2/layouts/collection (Duration: 56294.5ms | Allocations: 42620007)
2024-02-28T01:50:46 [I|app|a77bc44e] Rendered layout /usr/share/gems/gems/katello-4.9.0.21/app/views/katello/api/v2/layouts/collection.json.erb (Duration: 56295.7ms | Allocations: 42620229)
2024-02-28T01:50:46 [I|app|a77bc44e] Completed 200 OK in 56308ms (Views: 55209.9ms | ActiveRecord: 1090.2ms | Allocations: 42625491)

— Additional comment from on 2024-02-28T07:11:52Z

Applied the patch https://github.com/Katello/katello/pull/10865 and the result is:


grep c6009b28 /var/log/foreman/production.log
2024-02-28T02:08:15 [I|app|c6009b28] Started GET "/katello/api/v2/content_views?organization_id=1&nondefault=true&include_permissions=true&sort_by=name&sort_order=asc&per_page=50&page=1" for 10.64.136.25 at 2024-02-28 02:08:15 -0500
2024-02-28T02:08:15 [I|app|c6009b28] Processing by Katello::Api::V2::ContentViewsController#index as JSON
2024-02-28T02:08:15 [I|app|c6009b28] Parameters: {"organization_id"=>"1", "nondefault"=>"true", "include_permissions"=>"true", "sort_by"=>"name", "sort_order"=>"asc", "per_page"=>"50", "page"=>"1", "api_version"=>"v2"}

2024-02-28T02:08:41 [I|app|c6009b28] Rendered /usr/share/gems/gems/katello-4.9.0.21/app/views/katello/api/v2/content_views/index.json.rabl within katello/api/v2/layouts/collection (Duration: 25293.0ms | Allocations: 23362867)
2024-02-28T02:08:41 [I|app|c6009b28] Rendered layout /usr/share/gems/gems/katello-4.9.0.21/app/views/katello/api/v2/layouts/collection.json.erb (Duration: 25293.8ms | Allocations: 23363099)
2024-02-28T02:08:41 [I|app|c6009b28] Completed 200 OK in 25333ms (Views: 24960.5ms | ActiveRecord: 343.3ms | Allocations: 23389276)

It's improved but 25333ms still seems a bit slow. @jeremylenz, it seems like this is the best we can do for now, right?

— Additional comment from on 2024-02-28T14:51:30Z

I deleted all audit records and it loads pretty quick..Don't know if people could use that as a workaround while we get a fix in. It's definitely related to a change we added in 6.14 around auditing CV publishes but the querying for that needs to be improved based on number of audits we are looking at. Ex: 204600 on the repro before I cleared it..

— Additional comment from on 2024-02-28T22:49:44Z

(In reply to Samir Jha from comment #6)
> I deleted all audit records and it loads pretty quick..Don't know if people
> could use that as a workaround while we get a fix in. It's definitely
> related to a change we added in 6.14 around auditing CV publishes but the
> querying for that needs to be improved based on number of audits we are
> looking at. Ex: 204600 on the repro before I cleared it..

Nice finding. I'll ask the customer to apply. `foreman-rake audits:expire days=7` might be enough I think.

— Additional comment from on 2024-03-04T09:47:45Z

Hello,

Can we consider to make public this workaround suggested:

~~~
foreman-rake audits:expire days=7
~~~

From the case handled by Matt Jia, seems the cu resolved the issue.

— Additional comment from on 2024-03-04T13:13:37Z

As a workaround, you can use this command as stated in Comment 6:

~~~
foreman-rake audits:expire days=7
~~~

— Additional comment from on 2024-03-04T13:17:04Z

Ian, Adam, I wonder if we should investigate this issue a bit more, around Content Views and Audits.

— Additional comment from on 2024-03-08T20:33:23Z

The upstream issue: https://projects.theforeman.org/issues/36976 has a PR merged: https://github.com/Katello/katello/pull/10911

Updating BZ.

QE Tracker for https://issues.redhat.com/browse/SAT-24959
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2279313