Rationale

• It's a giant (41.300 LoC) C program in a security critical position: It multiplexes parallel sessions with lots of user-supplied data processing. Any vulnerability in it or its C libraries would make it possible to tamper with another Cockpit session.
• It only supports HTTP 1, requiring us to bundle our pages
• It implements a web server from scratch, and while the code is mature after 10 years, we still keep getting bug reports. We are also not keeping up with best web practices such as cookie restrictions.

Requirements

• A much smaller, maintainable, and understandable web server which only implements our Cockpit specific magic sauce (authentication, websocket protocol, session connection), and leaves the standard HTTP to a well-established library.
• HTTP 2/3 support so that we can stop bundling our pages. This gives much better caching/bandwidth efficiency and also makes our build chain easier.
• Implement the web server in a memory safe language as we cannot afford vulnerabilities there. Candidates are Python and Rust.

Design

The current idea is to take a page out of cockpit-desktop's book and try to move the bulk of the code into a per-session webserver (albeit still on the bastion host side, as we can't tunnel it easily through SSH). I.e.:

• a single top-level "proxy" endpoint. It starts from cockpit-tls, but also needs to do the authentication part and serve static content (login page, fonts, shared npm modules) for efficiency. We can afford to do that in a memory-safe language. That must support HTTP 2/3.
• a per-session ws instance which connects the web socket/bridge, and does the more intricate bits such as rewriting HTML (inserting our translations and such). This can probably support HTTP 1 only.

We have some initial code for experimentation: https://github.com/cockpit-project/cockpit/pull/20706 and https://piware.de/tmp/mockpit.py

Language

• [afedorova] No Rust expertise on the team, and no readiness for the Rust toolchain
• [pitti] I’m a big fan of Rust, which might influence my judgement. Still, I believe that at least prototyping and investigating the architecture is better done in Python, and only if we can demonstrate it that it is a bottleneck, consider Rust (the distro build chains still freak me out).
• [pitti] Especially with libuv-based runners, async Python is crazily (for a half-interpreted language) fast, let's not underestimate it. These are able to serve tens of thousands of parallel clients and 20K requests/s, we don't require anywhere near that performance. The bottleneck will always be I/O and network speeds. We had similar concerns about the Python bridge and it turned out to be completely unnoticeable.
• [Freya] Rust isn’t super complicated IMO but it is still a lot lower-level than Python and JS which we are used to. I do like when performance optimum languages like Rust are used instead of Python for things that would benefit greatly from it

With Python, we most probably want Starlette or even FastAPI (provides input validation and other goodies). Both are packaged in Fedora/Debian/Arch (but probably not RHEL, need to bundle there?). Lis likes aiohttp, but that only supports HTTP 1.

Idea: Service worker

Service workers are a very interesting concept:

• It would allow us (sort of) to move the whole "web server" part to running in the browser itself, and be very minimal. Between browser and the cockpit endpoint we'd already speak our JSON protocol, over a single connection.
• Caching should happen before in the browser as usual; query to the service worker only happens if document isn’t in the cache (ServiceWorker is just a proxy).
• Big caveat: ServiceWorker does not work in Firefox private windows; workaround is Containers, or enabling the “use http for testing” option; check Chromium
• Unknown: Do service workers have access to the TLS client certificate, so that we can continue to map them to users in cockpit-session for smart card support? If not, then this idea is dead.
• Service workers don’t work in sandboxed iframes

Let’s spend a few days doing a demo with a “mockpit” that just sends two or three simple string documents; validate our assumptions above and give us a feel for how this looks like.