Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-1758

Race condition in HttpServerExchange.state

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • None

      I have a simple handler which, essentially, does this:

      override def handleRequest(exchange: HttpServerExchange): Unit = {
       val out: F[Nothing, Unit] = for {
       out <- result(exchange).sandbox.leftMap(_.toEither)
       .redeemPure[TransportResponse](
       f => TransportResponse.Failure(handler.toRemote(exchange)(f)),
       v => TransportResponse.Success(v)
       )
       (code, headers, body) = convention.mapResponse(out)
       _ <- writeResponse(exchange, headers, code, body)
       _ <- F.sync(exchange.endExchange())
       } yield ()
      
       exchange.dispatch(new Runnable {
       override def run(): Unit = {
       BIORunner[F].unsafeRunAsyncAsEither(out)(errHandler.handle(RuntimeErrorHandler.Context.HttpRequest(exchange)))
       }
       })
       ()
      }

      The code in the `run` method runs my business logic in a separate thread pool and awaits until it exits.

      My `writeResponse` method looks like so:

      private def writeResponse(exchange: HttpServerExchange, headers: Map[HttpString, String], code: Int, json: Json): F[Nothing, Unit] = {
        for {
          response <- F.sync(json.printWith(printer))
          _ <- undertowEventListener.onHttpResponse(response, code, headers.map(kv => (kv._1.toString, kv._2)))
          _ <- F.sync {
            exchange.setStatusCode(code)
            headers.foreach {
              case (h, v) =>
                exchange.getResponseHeaders.add(h, v)
            }
            exchange.getResponseSender.send(response)
          }
        } yield {
        }
      
      }
      

      I don't have any other code in my app interacting with HttpServerExchange. Note that `writeResponse` runs in my thread pool.

       Everything worked well until I upgraded my hardware to Ryzen 3970x. Now undertow likes to throw here:

      public HttpServerExchange setStatusCode(final int statusCode) {
          if (statusCode < 0 || statusCode > 999) {
              throw new IllegalArgumentException("Invalid response code");
          }
          int oldVal = state;
          if (allAreSet(oldVal, FLAG_RESPONSE_SENT)) {
              throw UndertowMessages.MESSAGES.responseAlreadyStarted();
          }
      ...
      

      In case I run my code with JVM debugger attached - it never throws. For me it looks like a race condition, probably caused by thread-unsafe nature of HttpServerExchange#state field - as I said before my logic which completes the request runs in as a separate task in my threadpool.

       I think I wouldn't be able to implement an isolated repo - the issue manifests only on a Threadripper and it doesn't happen on every request (like 1 of 10). Nor I can come up with an exact scenario (yet), HttpServerExchange is extremely over-complicated and seems like I would need a lot of time to analyze the issue.

      Could you help?

              rhn-cservice-bbaranow Bartosz Baranowski
              pshirshov@gmail.com Pavel Shirshov (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: