Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-17075

Syndesis-prometheus pod is in error state after upgrade to Fuse Online 7.9 on OCP 3.11

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • fuse-7.9-GA
    • fuse-7.9-GA
    • Fuse Online
    • None

    Description

      When I upgrade Fuse Online 7.8 to Fuse Online 7.9, the syndesis-prometheus pod is in error state and contains the following error in the log:

      [mkralik@fedora installScript]$ oc logs -f syndesis-prometheus-1-jcfzm
      level=warn ts=2021-07-27T13:44:50.548Z caller=main.go:311 deprecation_notice="'storage.tsdb.retention' flag is deprecated use 'storage.tsdb.retention.time' instead."
      level=info ts=2021-07-27T13:44:50.548Z caller=main.go:360 msg="Starting Prometheus" version="(version=2.23.0, branch=rhaos-4.7-rhel-8, revision=9f61b39dc27b9125fab77359932cd64371c7bdb2)"
      level=info ts=2021-07-27T13:44:50.548Z caller=main.go:365 build_context="(go=go1.15.7, user=root@2d4ab9c99079, date=20210707-05:57:09)"
      level=info ts=2021-07-27T13:44:50.548Z caller=main.go:366 host_details="(Linux 3.10.0-1062.el7.x86_64 #1 SMP Thu Jul 18 20:25:13 UTC 2019 x86_64 syndesis-prometheus-1-jcfzm (none))"
      level=info ts=2021-07-27T13:44:50.548Z caller=main.go:367 fd_limits="(soft=1048576, hard=1048576)"
      level=info ts=2021-07-27T13:44:50.548Z caller=main.go:368 vm_limits="(soft=unlimited, hard=unlimited)"
      level=error ts=2021-07-27T13:44:50.550Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=data/queries.active err="open data/queries.active: permission denied"
      panic: Unable to create mmap-ed active query log
      
      goroutine 1 [running]:
      github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x29f831f, 0x5, 0x14, 0x30df0c0, 0xc00074d590, 0x30df0c0)
      	/go/src/github.com/prometheus/prometheus/promql/query_logger.go:117 +0x4cf
      main.main()
      	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:396 +0x53cc
      

      This is happening only on OCP 3.11 and only for upgrade. The clean installation works correctly. Also, installation and upgrade works correctly on OCP 4.7.

      I noticed that the `registry.redhat.io/openshift4/ose-prometheus:v4.7` image is used.

      So probably, the issue is in upgrade from `registry.redhat.io/openshift3/prometheus:v3.9` (Fuse Online 7.8) to `registry.redhat.io/openshift4/ose-prometheus:v4.7` (Fuse Online 7.9).

      Also the question is if the `ose-prometheus:v4.7` is supported on OCP 3.11 (it works with clean installation)

      Attachments

        Activity

          People

            cmiranda@redhat.com Claudio Miranda
            mkralik@redhat.com Matej Kralik
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: