Uploaded image for project: 'Maistra'
  1. Maistra
  2. MAISTRA-1779

ossm 2.0 istiod EDS Push error Distribution Event Queue overwhelmed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • maistra-2.0.0
    • maistra-2.0.0
    • envoy, istio
    • None
    • MAISTRA 2.0.0

      Build Info:
      OCP 4.5.5
      OSSM latest-2.0-qe

      How to produce:
      1. install SMCP 2.0 on OCP 4.5.5
      2. Deploy bookinfo sample app
      3. Apply Virtual Service changes to bookinfo app
      4. Visit bookinfo productpage several times
      5. Check SMCP pod log : istiod-basic-install pod discovery container log

      When we test bookinfo app with OSSM 2.0 on OCP, We see an error in the istiod-basic-install pod discovery container log:

      2020-08-24T20:15:20.399898Z info ads EDS: PUSH for node:ratings-v1-79d7d4f7bc-5m6k5.bookinfo clusters:21 endpoints:21 empty:0
      2020-08-24T20:15:20.401669Z error status Distribution Event Queue overwhelmed, status will be invalid.
      2020-08-24T20:15:20.401695Z error status Distribution Event Queue overwhelmed, status will be invalid.
      2020-08-24T20:15:20.402967Z error status Distribution Event Queue overwhelmed, status will be invalid.
      2020-08-24T20:15:20.404963Z error status Distribution Event Queue overwhelmed, status will be invalid.
      2020-08-24T20:15:21.732469Z info ads Push Status: {
      "pilot_eds_no_instances": {
      "outbound|9080||productpage.bookinfo.svc.cluster.local": {}
      },
      "pilot_no_ip": {
      "details-v1-5c9b9567cf-9w2h5.bookinfo":

      { "proxy": "details-v1-5c9b9567cf-9w2h5.bookinfo" }

      ,
      "productpage-v1-5b4b8f689c-znwbt.bookinfo":

      { "proxy": "productpage-v1-5b4b8f689c-znwbt.bookinfo" }

      ,
      "ratings-v1-79d7d4f7bc-5m6k5.bookinfo":

      { "proxy": "ratings-v1-79d7d4f7bc-5m6k5.bookinfo" }

      ,
      "reviews-v1-8446d565c7-qjcv8.bookinfo":

      { "proxy": "reviews-v1-8446d565c7-qjcv8.bookinfo" }

      ,
      "reviews-v2-5bbccc8b7-hr465.bookinfo":

      { "proxy": "reviews-v2-5bbccc8b7-hr465.bookinfo" }

      ,
      "reviews-v3-69d7ff8b5-zrslt.bookinfo":

      { "proxy": "reviews-v3-69d7ff8b5-zrslt.bookinfo" }

      }
      }
      2020-08-24T20:15:24.377284Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.385967Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.386042Z info ads ADS: "10.131.0.166:35782" sidecar~10.131.0.166~details-v1-5c9b9567cf-9w2h5.bookinfo~bookinfo.svc.cluster.local-73 terminated rpc error: code = Canceled desc = context canceled
      2020-08-24T20:15:24.509252Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.533427Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.533507Z info ads ADS: "10.131.0.167:55246" sidecar~10.131.0.167~ratings-v1-79d7d4f7bc-5m6k5.bookinfo~bookinfo.svc.cluster.local-75 terminated rpc error: code = Canceled desc = context canceled
      2020-08-24T20:15:24.562627Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.589513Z info ads ADS: "10.129.2.81:57956" sidecar~10.129.2.81~reviews-v1-8446d565c7-qjcv8.bookinfo~bookinfo.svc.cluster.local-78 terminated rpc error: code = Canceled desc = context canceled
      2020-08-24T20:15:24.589575Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.621239Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.631793Z info ads ADS: "10.131.0.168:50236" sidecar~10.131.0.168~reviews-v2-5bbccc8b7-hr465.bookinfo~bookinfo.svc.cluster.local-74 terminated rpc error: code = Canceled desc = context canceled
      2020-08-24T20:15:24.631848Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.656620Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:24.663143Z info ads ADS: "10.128.2.92:60176" sidecar~10.128.2.92~reviews-v3-69d7ff8b5-zrslt.bookinfo~bookinfo.svc.cluster.local-77 terminated rpc error: code = Canceled desc = context canceled
      2020-08-24T20:15:24.663193Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:25.310798Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
      2020-08-24T20:15:25.318118Z info ads ADS: "10.128.2.93:41568" sidecar~10.128.2.93~productpage-v1-5b4b8f689c-znwbt.bookinfo~bookinfo.svc.cluster.local-76 terminated rpc error: code = Canceled desc = context canceled
      2020-08-24T20:15:25.318190Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"

              yuaxu@redhat.com Yuanlin Xu
              yuaxu@redhat.com Yuanlin Xu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: