Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-42718

The ForemanInventoryUpload::Async::SingleHostReportJob, job always fails on Satellite 6.18.2 when IoP is enabled

    • Known Issue
    • Hide
      Insights On-Prem is enabled, but the uploader.sh script contains an incorrect URL, causing tasks to fail uploading reports to Insights On-Prem. To work around this problem, delete the file /var/lib/foreman/red_hat_inventory/uploader.sh, and it will be regenerated on the next inventory upload with the correct IoP url. As a result, the paused tasks will be resolved. Example command: `sudo rm /var/lib/foreman/red_hat_inventory/uploader.sh`.
      Show
      Insights On-Prem is enabled, but the uploader.sh script contains an incorrect URL, causing tasks to fail uploading reports to Insights On-Prem. To work around this problem, delete the file /var/lib/foreman/red_hat_inventory/uploader.sh, and it will be regenerated on the next inventory upload with the correct IoP url. As a result, the paused tasks will be resolved. Example command: `sudo rm /var/lib/foreman/red_hat_inventory/uploader.sh`.
    • In Progress
    • Manual

      With Insights On-Prem enabled and providing two services e.g., Advisor and Vulnerability, within Satellite, As new job \ task ForemanInventoryUpload::Async::SingleHostReportJob, was designed and this task is triggered automatically whenever a new Package Profile data is uploaded from a host to satellite. 

      In Satellite 6.18.2, when IoP is enabled, this task seems to always fail. 

       

      Affected Version:

      Red Hat Satellite 6.18.2 

      foreman_rh_cloud 12.2.12 and .13

       

      Steps to reproduce:

      • Install a satellite 6.18 ( connected ) and import a manifest into it
      • Don't register the system to insights\lightspeed
      • Enable basic repos and sync
      • Enable IoP
      • RESTART SATELLITE SERVICES
      • Register a system with satellite 
      • See if that triggers a `Single-host report job for host XXX` or not  in Monitor -> Satellite tasks --> tasks page
      • If not, Update or Downgrade some packages on the client host followed by the execution of `dnf uploadprofile --force-upload` command and then that should triggers a `Single-host report job for host XXX` 
      • Monitor the task in Monitor -> Satellite tasks --> tasks page and also follow /var/log/foreman/production.log 

       

      Actual Behavior:

      The task execution starts and it manages to run these two steps successfully

      ForemanInventoryUpload::Async::GenerateHostReport

      ForemanInventoryUpload::Async::QueueForUploadJob

       

      But then at 

      ForemanInventoryUpload::Async::UploadReportJob

      the task suspends. 

      In production.log we get to see that The task retries after every 1 minute ( thrice ) and fails everytime with error like this:

      2026-01-13T18:37:24 [I|app|099f1092] Adding slice 5dab99c7-a1bd-4a88-a381-91668fda6aa9
      2026-01-13T18:37:25 [I|app|099f1092] slice 5dab99c7-a1bd-4a88-a381-91668fda6aa9 was created with 1 hosts
      2026-01-13T18:37:25 [I|app|099f1092] Report generation finished
      2026-01-13T18:37:25 [I|app|099f1092] Archiving generated report
      2026-01-13T18:37:25 [I|app|099f1092] tar: ./
       099f1092 | ./metadata.json
       099f1092 | ./5dab99c7-a1bd-4a88-a381-91668fda6aa9.json
       099f1092 | 
      2026-01-13T18:37:25 [I|app|099f1092] Report archived successfully
      2026-01-13T18:37:26 [W|bac|099f1092] Polling failed, attempt no. 1, retrying in 59
      2026-01-13T18:37:26 [W|bac|099f1092] ERF42-9245 [Foreman::Exception]: Process exited with an unknown status: pid 108297 exit 22 (Foreman::Exception)
       099f1092 | /usr/share/gems/gems/foreman_rh_cloud-12.2.12/lib/foreman_inventory_upload/async/shell_process.rb:73:in `assert_task_status'
       099f1092 | /usr/share/gems/gems/foreman_rh_cloud-12.2.12/lib/foreman_inventory_upload/async/shell_process.rb:31:in `try_execute'
       099f1092 | /usr/share/gems/gems/foreman_rh_cloud-12.2.12/lib/foreman_inventory_upload/async/upload_report_job.rb:40:in `block in try_execute'
       099f1092 | /usr/share/ruby/tempfile.rb:358:in `create'
       099f1092 | /usr/share/gems/gems/foreman_rh_cloud-12.2.12/lib/foreman_inventory_upload/async/upload_report_job.rb:35:in `try_execute'
       099f1092 | /usr/share/gems/gems/foreman_rh_cloud-12.2.12/lib/foreman_rh_cloud/async/exponential_backoff.rb:37:in `poll_external_task'

       

      and finally stops with an error result.

      The exact same thing happens if i go to Administer --> Inventory Upload page and trigger and Generate and Upload job from there i.e. ForemanInventoryUpload::Async::GenerateReportJob

      This job task also have an action for ForemanInventoryUpload::Async::UploadReportJob and that fails in the exact same way

       

      Expected Behavior:

      No such issues

       

      Additional Info:

      Go to Administer --> Inventory Upload page, expand the Organization , Go to Uploading tab and see there are two upload attempts happening:

      1. 

      > content-length: 8839
      > content-type: multipart/form-data; boundary=------------------------3e57737cebdf7ccd
      >
      * TLSv1.2 (OUT), TLS header, Unknown (23):
      } [5 bytes data]
      * We are completely uploaded and fine
      

       

      2. 

      * The requested URL returned error: 401
      * TLSv1.2 (OUT), TLS header, Unknown (23):
      } [5 bytes data]
      * stopped the pause stream!
      * Connection #0 to host cert.cloud.redhat.com left intact
      curl: (22) The requested URL returned error: 401
      

       

      this second attempt fails and perhaps curl exits with error 22 , which is what we see in the production.log as well 

      2026-01-13T18:37:26 [W|bac|099f1092] Polling failed, attempt no. 1, retrying in 59
      2026-01-13T18:37:26 [W|bac|099f1092] ERF42-9245 [Foreman::Exception]: Process exited with an unknown status: pid 108297 exit 22 (Foreman::Exception)

      For a user with 1000s of hosts and Sat 6.18 with IoP enabled, would result into 1000s of paused tasks ( because of this issue )

       

      Testing Steps

       

      • Install a satellite 6.18 ( connected Non IoP ) and import a manifest into it
      • Enable basic repos and sync
      • Register at least one  insights enabled host
      • Ensure satellite generates at least one upload to insights. Verify  uploader.sh file exists at (/var/lib/foreman/red_hat_inventory)
      • Enable IoP
      • Add firewall rule to not connect to cloud.redhat.com 
      • RESTART SATELLITE SERVICES
      • Update or Downgrade some packages on the client host followed by the execution of `dnf uploadprofile --force-upload` command and then that should triggers a `Single-host report job for host XXX` 
      • Monitor the task in Monitor -> Satellite tasks --> tasks page and also follow /var/log/foreman/production.log 
      • Make sure that we are not calling cloud.redhat.com 

       

       

       

              rhn-engineering-jlenz Jeremy Lenz
              rhn-support-saydas Sayan Das
              Ladislav Vasina Ladislav Vasina
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: