Uploaded image for project: 'OpenJDK'
  1. OpenJDK
  2. OPENJDK-495

Cryostat can OOM when re-uploading JFR recordings into archives

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • Cryostat 2.0
    • None
    • None
    • False
    • False
    • Workaround Exists
    • Hide

      When Cryostat is deployed in the supported scenario by using the Cryostat Operator, users can bypass the POST /api/v1/recordings HTTP handler and copy their .jfr binary data directly into the Cryostat archives in the OpenShift PVC.

       

      Use `oc project` to determine the deployment namespace, `oc get pods` to determine the Cryostat pod name, and `oc get cryostats` to determine the Cryostat container name.

       

      oc exec -i -n $MYNAMESPACE -c $CRYOSTAT_CONTAINER_NAME $CRYOSTAT_POD_NAME – mkdir /opt/cryostat.d/recordings.d/unlabelled # this may fail if the required "unlabelled" directory already exists. If so, proceed regardless

      oc cp vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr $CRYOSTAT_POD_NAME:/opt/cryostat.d/recordings.d/unlabelled/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr -c $CRYOSTAT_CONTAINER_NAME

      After copying the recording directly into the PVC archives location, we can verify that Cryostat is aware of the uploaded file using httpie, curl, or wget:

      $ https GET https://cryostat-sample-myproject.apps-crc.testing/api/v1/recordings Authorization:"Bearer $(oc whoami -t)"
      HTTP/1.1 200 OK
      Set-Cookie: ac6fe4a79b8104f3ee178c2547bbced6=b010120f48b63047a41c7183900059fc; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private
      content-encoding: gzip
      content-length: 182
      content-type: text/plain
      set-cookie: ac6fe4a79b8104f3ee178c2547bbced6=b010120f48b63047a41c7183900059fc; path=/; HttpOnly; Secure; SameSite=None

      [

      { "downloadUrl": "https://cryostat-sample-myproject.apps-crc.testing:443/api/v1/recordings/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr", "name": "vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr", "reportUrl": "https://cryostat-sample-myproject.apps-crc.testing:443/api/v1/reports/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr" }

      ]

      Show
      When Cryostat is deployed in the supported scenario by using the Cryostat Operator, users can bypass the POST /api/v1/recordings HTTP handler and copy their .jfr binary data directly into the Cryostat archives in the OpenShift PVC.   Use `oc project` to determine the deployment namespace, `oc get pods` to determine the Cryostat pod name, and `oc get cryostats` to determine the Cryostat container name.   oc exec -i -n $MYNAMESPACE -c $CRYOSTAT_CONTAINER_NAME $CRYOSTAT_POD_NAME – mkdir /opt/cryostat.d/recordings.d/unlabelled # this may fail if the required "unlabelled" directory already exists. If so, proceed regardless oc cp vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr $CRYOSTAT_POD_NAME:/opt/cryostat.d/recordings.d/unlabelled/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr -c $CRYOSTAT_CONTAINER_NAME After copying the recording directly into the PVC archives location, we can verify that Cryostat is aware of the uploaded file using httpie, curl, or wget: $ https GET https://cryostat-sample-myproject.apps-crc.testing/api/v1/recordings Authorization:"Bearer $(oc whoami -t)" HTTP/1.1 200 OK Set-Cookie: ac6fe4a79b8104f3ee178c2547bbced6=b010120f48b63047a41c7183900059fc; path=/; HttpOnly; Secure; SameSite=None cache-control: private content-encoding: gzip content-length: 182 content-type: text/plain set-cookie: ac6fe4a79b8104f3ee178c2547bbced6=b010120f48b63047a41c7183900059fc; path=/; HttpOnly; Secure; SameSite=None [ { "downloadUrl": "https://cryostat-sample-myproject.apps-crc.testing:443/api/v1/recordings/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr", "name": "vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr", "reportUrl": "https://cryostat-sample-myproject.apps-crc.testing:443/api/v1/reports/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr" } ]
    • Hide

      pushd $(mktemp -d)

      git clone --depth 1 https://github.com/cryostatio/cryostat

      cd cryostat

      CRYOSTAT_IMAGE=registry.redhat.io/cryostat-20-tech-preview/cryostat-rhel8:2.0.0-2.1634741315 sh run.sh

       

      In another terminal, after waiting a moment for the Cryostat container to start up,

      curl -kv --progress-bar -F recording=@vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr https://0.0.0.0:8181/api/v1/recordings | cat

       

      After curl finishes transferring the file to Cryostat, the Cryostat container should die and exit within a few seconds.

       

      I cannot attach my 156MB .jfr file tested with since it is too large. I collected it by running a Profiling recording with size limit of 150MB on the vertx-fib-demo sample app and letting that run for several hours, then coming back later to collect the sample. I'm not sure how long it actually takes for the recording to accumulate to that size before the size limit takes effect.

      Show
      pushd $(mktemp -d) git clone --depth 1 https://github.com/cryostatio/cryostat cd cryostat CRYOSTAT_IMAGE=registry.redhat.io/cryostat-20-tech-preview/cryostat-rhel8:2.0.0-2.1634741315 sh run.sh   In another terminal, after waiting a moment for the Cryostat container to start up, curl -kv --progress-bar -F recording=@vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006T152006Z.jfr https://0.0.0.0:8181/api/v1/recordings | cat   After curl finishes transferring the file to Cryostat, the Cryostat container should die and exit within a few seconds.   I cannot attach my 156MB .jfr file tested with since it is too large. I collected it by running a Profiling recording with size limit of 150MB on the vertx-fib-demo sample app and letting that run for several hours, then coming back later to collect the sample. I'm not sure how long it actually takes for the recording to accumulate to that size before the size limit takes effect.

    Description

      Cryostat 2.0.0 consumes more memory than expected when a client sends a well-formed request to POST /api/v1/recordings. When deployed in a container with a 512MB memory limit, which is a fairly standard default limit, Cryostat 2.0.0 can be made to reliably OOM by simply uploading a ~150MB or larger JFR binary recording file.

      Attachments

        Activity

          People

            aazores@redhat.com Andrew Azores
            aazores@redhat.com Andrew Azores
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: