When reviewing a PR for swatch-contracts changes, we started to encounter HTTP responses from the upstream IT Partner Gateway service.  swatch-contracts would respond with a 500 error that said "Api response has status code 403" and have no evidence of an error in the logs.  Switching branches, cleaning, rebuilding, etc. revealed that sometimes the error did get captured in the logs, and the payload response was different.

After a bunch of trial and error and dumping stack traces, I narrowed it down to that sometimes the partnerApi client is throwing jakarta.ws.rs.ProcessingException exception (unexpected), while other times it's throwing com.redhat.swatch.clients.rh.partner.gateway.api.resources.ApiException (expected). 

When our openapi generator generates our Resource interfaces, all the methods include "throws ApiException, ProcessingException", so it's not completely unexpected - but ProcessingException falls through the cracks because it extends RuntimeException so the IDEs aren't always in our face telling us that exception needs to be handled.

What's interesting is that if you debug and inspect the ProcessingException, it's actually "aware" that it's an ApiException type and that's where it's getting its exception message from.  I tracked it down to  the resteasy-reactive quarkus library we're using is having a hard time wrapping and rethrowing along these lines. https://github.com/quarkusio/quarkus/blob/d353f20ec2f0e3697ed843a25810f6588be1584c/extensions/resteasy-reactive/rest-client-reactive/runtime/src/main/java/io/quarkus/rest/client/reactive/runtime/MicroProfileRestClientResponseFilter.java#L55 

I'm not entirely sure what's going on (library bug?) to cause ProcessingException bubbling up instead of ApiException, but in the meantime I think we should at least throw out a catch block and handle it in ContractService.java.  A lot of time was lost reviewing that PR referenced in the beginning because there was very little evidence of what API call was failing and why.

Steps to reproduce:

Note: The ability to reproduce this seems to happen at compile time and seemingly random as to why.  When making the curl request, if you don't immediately reproduce the issue, you're going to have to stop the application, try cleaning and recompiling (maybe even switch some branches), and starting it up again.

Make sure that you don't have any KEYSTORE env vars set, and no env vars pointing the partner api gateway URL to a mock.  We're intentionally trying to have swatch-contracts fail authentication to get the error.  I haven't tried intentionally mocking a 403 response yet.

unset SWATCH_INTERNAL_SUBSCRIPTION_ENDPOINT
unset KEYSTORE_RESOURCE
unset ENTITLEMENT_GATEWAY_URL
 

./gradlew clean :swatch-contracts:quarkusDev 

curl -X POST --location "http://localhost:8000/api/swatch-contracts/internal/rpc/partner/contracts" \
    -H "x-rh-swatch-psk: placeholder" \
    -H "Content-Type: application/json" \
    -d "{
  \"action\": \"contract-updated\",
  \"redHatSubscriptionNumber\": \"12400374\",
  \"currentDimensions\": [
    {
      \"dimensionName\": \"cpu-hours\",
      \"dimensionValue\": \"5\",
      \"expirationDate\": \"2018-02-10T09:30Z\"
    },
    {
      \"dimensionName\": \"instance-hours\",
      \"dimensionValue\": \"10\",
      \"expirationDate\": \"2018-02-10T09:30Z\"
    }
  ],
  \"cloudIdentifiers\": {
    \"awsCustomerId\": \"HSwCpt6sqkC\",
    \"awsCustomerAccountId\": \"795061427196\",
    \"productCode\": \"1234567890abcdefghijklmno\"
  }
}"