Monitoring console plugin should avoid browser-caching failures

Description of problem:

MON-2967 and cmo#1890 moved the Observe console menu into a console plugin (in 4.15? 4.14?). Sometimes If-Modified-Since browser caching results in failures that result in a missing Observe menu, and when the user eventually finds /k8s/cluster/operator.openshift.io~v1~Console/cluster/console-plugins, render failure as:

Failed to get a valid plugin manifest from /api/plugins/monitoring-plugin/
SyntaxError: Unexpected end of JSON input

This appears to be the result of the browser's If-Modified-Since caching:

$ curl -sH Accept:application/json -H Cache-Control:max-age=0 -H 'Cookie: openshift-session-token=...; login-state=...; ...; csrf-token=...'
-H 'If-Modified-Since: Fri, 03 Nov 2023 00:47:45 GMT' -i https://console.build02.ci.openshift.org/api/plugins/monitoring-plugin/plugin-manifest.json
HTTP/1.1 200 OK
date: Tue, 21 Nov 2023 16:52:55 GMT
etag: "65444331-9a2"
last-modified: Fri, 03 Nov 2023 00:47:45 GMT
referrer-policy: strict-origin-when-cross-origin
server: nginx/1.20.1
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-frame-options: DENY
x-xss-protection: 1; mode=block
content-length: 0

While a more recent If-Modified-Since returns populated JSON:

$ curl -sH Accept:application/json -H 'If-Modified-Since: Fri, 10 Nov 2023 10:47:45 GMT' -H 'Cookie: openshift-session-token=...; login-state=...; ...; csrf-token=...' https://console.build02.ci.openshift.org/api/plugins/monitoring-plugin/plugin-manifest.json | jq . | head
{
  "name": "monitoring-plugin",
  "version": "1.0.0",
  "displayName": "OpenShift console monitoring plugin",
  "description": "This plugin adds the monitoring UI to the OpenShift web console",
  "dependencies": {
    "@console/pluginAPI": "*"
  },
  "extensions": [
    {

Disabling caching on the monitoring-plugin side would avoid this issues. But fixing 304 handling in the console's proxy would likely also resolve the issue.

Version-Release number of selected component (if applicable):

Seen in 4.15.0-ec.2. Reproduced in ec.2. Failed to reproduce in ec.1. Possibly a regression from ec.1 to ec.2, although I haven't identified a regressing commit yet.

How reproducible:

Seen multiple times by multiple users in 4.15.0-ec.2 in two long-lived clusters, and also reproduced in an ec.2 Cluster Bot cluster. Likely consistently reprodible on ec.2.

Steps to Reproduce:

1. Install a cluster, e.g. with launch 4.15.0-ec.2 gcp.
2. Log into the console and use the developer tab to get an openshift-session-token value from a successful HTTPS request.
3.

$ curl -ksi -H "Cookie: openshift-session-token=${TOKEN}" "https://${HOST}/api/plugins/monitoring-plugin/plugin-manifest.json" | grep 'HTTP\|content-\|last-modified'

with your ${TOKEN} and ${HOST}, to confirm 200 responses and find the last-modified value.
4.

$ curl -ksi -H "If-Modified-Since: ${LAST_MODIFIED}" -H "Cookie: openshift-session-token=${TOKEN}" "https://${HOST}/api/plugins/monitoring-plugin/plugin-manifest.json"

with your ${TOKEN}, ${HOST}, and ${LAST_MODIFIED}.

Actual results:

Observe menu is missing, with browser-console logs like:

Failed to get a valid plugin manifest from /api/plugins/monitoring-plugin/
SyntaxError: Unexpected end of JSON input

200 responses with no content when If-Modified-Since is greater than or equal to the content's last-modified.

Expected results:

Reliably successful loading of the monitoring console plugin, with a 304 when If-Modified-Since is greater than or equal to the content's last-modified.

Possibly more obvious warnings pointing at /k8s/cluster/operator.openshift.io~v1~Console/cluster/console-plugins when plugins fail to load.

Additional info:

Using the browser's development tools to disable caching while loading the console avoids the problematic caching interaction.