-
Bug
-
Resolution: Done
-
Normal
-
RHOAI_2.5.0
-
False
-
-
False
-
-
When kserve and modelmeh are running in the same namespace, modelmesh container show these errors:
{"instant":{"epochSecond":1701291690,"nanoOfSecond":966215444},"thread":"ll-elg-thread-2","level":"INFO","loggerName":"com.ibm.watson.modelmesh.ModelMesh","message":"Returning READY to readiness probe (did not find any other pods in terminating state)","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","contextMap":{},"threadId":35,"threadPriority":5} Nov 29, 2023 9:01:37 PM io.grpc.netty.NettyServerTransport notifyTerminated INFO: Transport failed io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET /stats/prometheus at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109) at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:317) at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247) at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
There are 3 networkpolicy in the namespace:
- allow-from-openshift-monitoring-ns
- istio-expose-route-minimal
- istio-mesh-minimal
If allow-from-openshift-monitoring-ns network policy is deleted, the error message is not showing up anymore. So I think this networkpolicy is the culprit of this issue. However, it is not 100% so it needs more debugging.
Reference:
https://github.com/orgs/opendatahub-io/projects/42?pane=issue&itemId=40292089
NOTE: This task is migrated from GitHub to Jira. See previous discussion on GH.